HudsonWindowsSlavesSetup

Created on: Aug 14, 2007 5:02 AM by Aleksandar Kostadinov - Last Modified: Jul 31, 2009 1:21 PM by Aleksandar Kostadinov

Overview

We want to use NFS to simplify storage management. As well ssh and not JNLP slaves that is under question if it is a good idea. Plus some additional configuration so we can start things with a limited and not administrator account. And some scripts to remount NFS on every start and make a slave to run from the same location as on UNIX, load common configuration (MCAST_ADDR, MY_TESTIPs, java options, etc.) and that is. As long as we use ssh that can/should be passed to ssh for execution so we have svn control over it and don't need to modify all the slaves for a change.

Used software

We use cygwin with password auth due to limitations of windows to authorize users without a password. As well ssh server is the only means found for now that doesn't line buffer on the server side. So telnet and winexe are not yet suitable and telnet will probably never be.

For the NFS client - MS SFU. It comes with nice unix utilities like kill. For Win 2003 R2 that should come bundled. As well we are supporting CIFS but not for workspace. As well WinR2k3 R2 slaves are only supported with local workspace because NFS locking doesn't work from privileged ports.

PSkill we use to kill stale java processes so have it in path (install pstools).

More Detailed Instructions

Create Hudson user (see Win2k3 R2 setup)

Create a limited user account named hudson
Make it a member of the backup Operators group (dunno if required for the case but otherwise it could be unable to start in the background)
on win2k3 allow cmd.exe to be executed by the hudson user (again dunno if required any more... it was required for cruisecontrol started by task scheduler on system start-up)
Be sure to login at least once as hudson so home directory gets created!
Set the password for the account to never expire and like the other win slaves

Install MS SFU

you can skip this and use the smb bridge as highly problematic

2008 and vista

don't support easy mapping so anon access only possible
had problems running java off NFS but no problems from CIFS

xp and 2000 don't support AFAICT
2003 R2 has problems with using priviledge ports for locking

see Win2k3 R2 setup below for more details after installation
Get installation binaries from qa_tools on the qa smb filer
install components - client and mapper + updates (see readme in the dir)
map users to unix uid numbers to match their home directories (probably only hudson user using passwd file)
use hard mount

Install cygwin sshd

I use instructions from here. I will briefly describe the process though.

Get the cygwin setup from http://www.cygwin.com/setup.exe.
Choose a mirror close to Boston - NY would be fine (mirror.nyi.net)
Select packages

From Admin, select all packages.
From Archive, select unzip and zip packages.
From Base, leave the default, select all packages.
From Doc, leave the default, man and 'cygwin doc' packages.
From Editors, select vim package.
From Net, select openssh (openssl will get checked automatically), rsync and tcp_wrappers packages.
Find and select wget, lynx, svn, cvs and whatever you like as well see soft needed on AddingSlaveToMasterInHudson!

Setup sshd

start from a cygwin shell "ssh-host-config"

note: for vista you need to be "Administrator" as other admin accounts don't work

let it do whatever it likes with privilege separation and install as a service (start sshd)
set CYGWIN=binmode tty ntsec
use mkpasswd utility to fill up /etc/passwd (mkpasswd -l > /etc/passwd)
don't bother setting up public key auth as we can't use that with network storage anyways...

Try ssh and if it works, then you are done!

Install pskill

download pstools (probably can be found on NFS)
unzip somewhere and set global PATH variable or put in system32
IMPORTANT Run pskill once when logged in as hudson. Then accept license agreement. If you don't do that hudson startup script will hang.

Start/stop scripts

I tricked you ;p no stop script.

The issue here is that we have to use ssh with password auth due to windows "security model". To be able to use password auth from script we have to trick ssh that it doesn't run from on a terminal and are using perl for the purpose. See the attached manually.txt for more information.

The scripts are already under subversion control, so ignore attached files. The local startSlave.bat is not used any more. Everything is on the network storage.

Procedure:

use "/qa/home/hudson/launchWinSshSlave.sh host conf_file " as startup command.
launchWinSshSlave.sh executes sshWin.sh
sshWin.sh accomplishes ssh connection using echo_pass.sh and setsid.pl then executes startSlave.bat
the conf file in launchWinSlave/conf directory features a command for ssh to execute on the remote host.
is list of features - actually batch files located in launchWinSlave/init.d used to setup environment. For example see config for win2k3 and XP below:

launchWinSshSlave.sh host1 nfs-sh allwin fixtemp javaloc mem1g nfslocat nfstools32 ips javaopts nfslaunch
launchWinSshSlave.sh host2 cifs-sh allwin fixtemp javaloc mem700m nfslocat nfstools32 ips javaopts nfslaunch

That creates a pretty flexible foundation for configuring new slaves needing just to write and reuse batch files in the init.d directory.

Slave configuration:

Besides startup command choose wisely working directory. It should be h:\hudson\hudson_workspace for current scripts on network storage aware windows slaves. Drive T is /qa/tools in case you need some utility (ant, java, etc) but these locations should provided in environment variables by the batch scripts executed.

Advanced Start-Up (you don't need that any more - see above):

You can't use quotes for the startup command so use as startup command "bash -c $ -- eval export MAX_JAVA_MEM=-Xmx700m; exec /qa/home/hudson/launchWinSlave.sh dev29;" to specify memory.

Non-network storage slaves

I use the same startup script as with network storage slaves. Differences:

Startup script begins with downloading slave.jar from the QA http server using wget from the cygwin installation
no mounting drives
use some local directory as a working directory
have utilities - java, ant, maven, svn, etc. locally

Useful commands

runas /user:mymachine\hudson cmd
runas /user:mymachine\cruisecontrol "explorer /separate"
runas /user:mymachine\cruisecontrol "C:\WINDOWS\system32\posix.exe /u /c /bin/ksh -l"

Links with old information

Win2k3 R2 detailed setup

   * windows update
   * setup ntp time sync if not yet configured "net time /setsntp:clock.corp.redhat.com"
   * install windows components (you need setup CD2)
      * Subsystem for UNIX-based Applications
      * Othen Network File and Print Services
         * exclude server for NFS
         * include User Name Mapping
   * install "Utilities and SDK for UNIX-based Applications" as suggested by setup
      * custom installation
      * everything except visual studio debugger seems potentially useful to me
      * I set setuid in case that functionality happens to be needed (we don't care about security too much)
      * don't choose case sensitive settings
   * create a user named "hudson" in a group hudsongroup
      * set the standard hudson pass
      * make member of these groups
         * Remote Desktop Users
         * TelnetClients
   * Start regedit (that's needed to mount when conns are allowed from privileged ports only)
      * HKLM\Software\Microsoft\Client for NFS\CurrentVersion\Default
      * Add UseReservedPorts DWORD value "1"
      * Restart the Client for NFS service to allow the change to take effect.
   * Still could't figure out how to do the same for nfslock
   * setup NFS client and User Name mapping
      * administrative tools -> Microsoft Services For Network File System
         * Right click -> properties on Client for NFS
            * Hard mount
            * from file permissions allow group write
         * Right click -> properties on User Name Mapping
            * Use password and group files
            * use c:\etc\ passwd and group as files
            * from simple mapping select use simple maps
         * Fill in passwd and group files with data for:
           root,cruisecontrol,test_cc,hudson
           you can copy/paste unix entries
         * mapping
            * expand User Name Mapping
            * right click user maps and group maps to enable "show simple maps"
            * right click user maps to create map of root user with jbossqa and Administrator
            * right click group maps to map hudson with hudsongroup and root with Administrators
   * enable SUA telnet (skip that)
      * start ksh from SUA utilities
      * vi /etc/inetd.conf
      * reboot or kill -HUP 
      * will be used instead of cygwin ssh when hudson above 1.164 installed with text mode slave
   * install cygwin
      * see above
   * add the private IPs to the network interface
   * set computer name and DNS suffix

Windows version specific notes

Windows 2k

This one is crap about mounts. Once you mount a drive in a session you can neither mount/use the same letter and network path in another session, nor umount the sessions. The only fix I've found is restart. So hudson start-up script should umount all shares with "net use * /delete /y".
As well I'm still not having the 2k machine work properly but made it somehow work with textmode slave.

oleksdovz

Шукати в цьому блозі