Editing Scanco Administration (section)

===Recover System===
====Consistently needs warm-up scanner - Shutter-failsave-Job====

If the scanner is consistently needing to warm-up on consecutive days, and was not powered off, this is likely due to the Shutter-failsave-Job disappearing from the 3309-Batch queue (for XT1). The job must be re-submitted. 

The executing file is located in the CALIB3309 ($ um:submit_shutter_test.com) and the command to submit the job is:
<pre>
$ submit/after="tomorrow"/noprint/que=uct_measurement_queue um:submit_shutter_test.com
</pre>

You can check the status by typing: 
<pre>
$ show que *3309_batch*/all
</pre>

This job tends to disappear now and then, perhaps because of an abrupt shutdown of the workstation or maybe when stop/pause/re-start the queue. Contact Scanco if this does not resolve.


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Disk not mounted/unavailable, directory not found, volume not enabled====
Disk is not mounted or is unavailable.  Error opening ...., directory not found, volume is not software enabled.
<pre>
%DIRECT-E-OPENIN, error opening DISK4:[MICROCT.PROJECTS]*.*;* as input
-RMS-E-DNF, directory not found
-SYSTEM-F-VOLINV, volume is not software enabled
</pre>
This can happen if communication is broken between workstations, causing the disk mount to time out.  Check if this is by showing the devices, then dismount and remount the disk.
<pre>
$ show dev

Device                  Device           Error    Volume         Free  Trans Mnt
 Name                   Status           Count     Label         Space Count Cnt
SC3309$DKA100:          Online               0
SC3309$DKD0:            Mounted              0  ALPHASYS        3.18GB   642   2
SC3309$DKD1:            Mounted              0  AUSER_01       42.84GB     4   2
SC3309$DKD2:            Mounted              0  ADATA_01      173.00GB     4   2
SC3309$DKD3:            Mounted              0  ADATA_02A     215.80GB     1   2
SC3309$DKD4:            Mounted              0  ADATA_02B     215.85GB     1   2
SC3309$DKD5:            Mounted              0  ADATA_02C     215.81GB     1   2
SC3309$DQA0:            Online               0
SC3309$DQA1:            Offline              1
SC3309$DQB0:            Offline              1
SC3309$DQB1:            Offline              1
SCEVAL$DKA0:            Online               0
SCEVAL$DKA1500:         Mounted              0  (remote mount)                 1
SCEVAL$DKC0: MntVerifyTimeout 0 ADATA_03A 541.62GB 1 2 
SCEVAL$DKC1: MntVerifyTimeout 0 ADATA_03B 541.38GB 1 2 
SCEVAL$DKC2: MntVerifyTimeout 0 ADATA_03C 541.60GB 1 2 
SCEVAL$DQA0:            Online               0


$ set host 0
system

$ dismount SCEVAL$DKC0:
$ mount/cluster SCEVAL$DKC0: ADATA_03A
%MOUNT-I-MOUNTED, ADATA_03A mounted on _SCEVAL$DKC0:
$ mount/cluster SCEVAL$DKC1: ADATA_03B
%MOUNT-I-MOUNTED, ADATA_03B mounted on _SCEVAL$DKC1:
$ mount/cluster SCEVAL$DKC2: ADATA_03C
%MOUNT-I-MOUNTED, ADATA_03C mounted on _SCEVAL$DKC2:
$
</pre>

If this does not restore the disk, then shutdown the computer.  At the blue screen...
<code>>>> init</code>
Wait for blue screen to return.
<code>>>> run bios pya0</code>
Press F2 at the prompt to accept potential data loss.
Wait for blue screen to return.
<code>>>></code>
Press power button to turn off.
Press power button to turn on.


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Kill hanging program====

If a program is hunging, it may not be necessary to shut down the system and restart. For example, if the <code>uct_evaluation</code>
program is hung, do the following

<pre>
$ show dev/full microct_scsi

$ stop/id=<Owner process ID> (Replace the <> with the actual ID)

for example: 

$ stop/id=2280091C 
</pre>


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Reset the display/GUI====
These commands will terminate the current login session and may restore a frozen or messed up screen.  You will need to remotely access the problem workstation either from another micro-ct workstation or using a personal computer.

Use the set host command to connect to the problem workstation (e.g., thomas) using the node name (e.g., sc3309) or the IP address (e.g., thomas.kin.ucalgary.ca). 

 $ set host sc3309
 login:  system
 password: 

Issue the reset display command.

 $ @sys$startup:decw$startup restart

At the prompt, type "y".


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Restart screen server====

If the monitor fails to show the login prompt and displays a blank gray screen...telnet to the computer from another computer (e.g. a PC or so) and login as user system. Run the restart command:

<pre>
$set host gordon.ucalgary.ca.    [or other computer name]
$username: system
$password: [system password]
$@sys$startup:decw$startup restart</pre>
confirm (answer "yes" at the prompt) and logout.
<pre>
$y
$logout</pre>

This will kill all programs on your screen, but it will leave all batch jobs running, so if you do a BATCH measurement, this will survive! (but interactive scan will be killed).

If you are restarting a monitor connected to a KVM box, unplug and re-plug in the box.

This command also helps when the mouse/keyboard freezes.


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Restart Alpha stations====

[[Image:Bones2.jpg|20px]] <b>Warning:</b> Do not do this unless absolutely necessary.

For shutdown

At the login screen, type "shutdown"/no password and hit enter. 
Then, you can turn the computer off at the prompt.

Then, you may turn the disk array off (unplug it as there is no power switch).

After you move the computer, power on the disk array and then the computer.


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Slow/non-responsive computer====

This is usually caused by people logging in via SSH multiple times.  The solution involves resetting the SSH/TCPIP daemon.

:*If you are local at the computer:

You will need to log in as user <code>system</code>.  ONLY EXPERIENCED USERS SHOULD EVER LOG IN AS SYSTEM!

The system will be extremely slow (it will take 5 minutes just to get a log in prompt).  This is expected.

<pre>
$ set host 0   ! user name 'system' and password	
$
$ @sys$startup:tcpip$ssh_shutdown    ! shutdown SSH
$
$ set host 0   ! user name 'system' and password	
$
$ @sys$startup:tcpip$ssh_startup     ! restart SSH
</pre>

Alternatively, you can reboot the system to reset the SSH.  This is not an appropriate solution and it is solving the problem by hitting it with a jack hammer.  Sometimes it is the only choice though.

:*If you are remotely logged in:

You will have to log in as user <code>system</code> and run a script called STOP_START_TCPIP.COM.  

This works essentially the same as for the above method, except that since it would be impossible to restart SSH remotely once it is shut down, two separate scripts are called and submitted on SCEVAL_FAST (make sure the queue isn't clogged) separated by a 1 minute delay. The first script called is STOP_TCPIP.COM and the second is START_TCPIP.COM.

To execute this process, you must log in as system, go to the SYS$SCRATCH directory, and interactively execute the script:
<pre>
$ @STOP_START_TCPIP.COM
</pre>


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Shutdown a workstation locally====
These instructions were provided by Pascal in May 2026:

    - end the session
    - enter 'shutdown' as username and hit enter key for a few times (no password needed)
    - the screen eventually turns black/blank
    - just give it another minute or so, then press and hold the power button until the workstation turns off
    - remove both power cords for a few minutes

    - reconnect power cords
    - wait for at least 30 seconds
    - press and release again the power button; the workstation will then turn on with a slight delay

The workstation should then be back online after a few minutes.

====Shutdown a workstation remotely====
If a workstation becomes unresponsive, it may be possible to shut it down remotely from another workstation or personal computer, which is preferable over doing a hard power off.  These instructions were provided by Mark Blanchard.

'''1. Remote shutdown from another micro-ct workstation'''.  MAY NOT WORK FOR CLUSTER SERVER

Login as the system user.

 $ set host 0
 user:  system
 password:

Start the system manager.
 $ mcr sysman

Set the environment to the workstation to be shutdown (e.g., thomas) using its node name (e.g., sc3309).
 SYSMAN>< set env/node=sc3309

Issue the shutdown command.  Add a time delay, e.g., 5 min, if you want to give users time to disconnect and the automatic reboot flag if you want to restart.
 SYSMAN> shutdown node /min=5 /automatic_reboot

*You need to remain logged into the session running the shutdown until the other node has been shutdown completely!!!

'''2. Remote shutdown from a personal computer'''.

Begin a remote session to the workstation and login as the system user.
 
Create a detached terminal.
 $ create/terminal/detach
 
On the PC, port 6000 needs to be accessible to the micro-ct workstation and in the X-server's configuration (e.g., in file X0.hosts for Xming). 
 
Issue the shutdown command.
 $ shutdown


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Slow computers - Change RAID battery for Itanium computers====

If computers become sluggish, particularly when saving data after a scan, have Scanco check the batteries in the RAID drive. The batteries in SCEVA3 were changed 2014-11-10. There are two batteries. Order the HP brand (should look like the ones in the image).
*Instructions to open the Itanium work station and access the PCI card: [[Media:HP_rx2660_PCI_card.pdf |'''Change PCI card''']].
*Instructions to change the batteries after accessing the PCI card: [[Media:HP_SA6402_Batteries.pdf |'''Change batteries''']].


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Frozen computer - "bootstrap failure" on boot up==== 
The micro-CT computer is frozen or there is a "bootstrap failure" on boot up, This happens to the client computers (those connected to the scanners) when there is interruption in the network.  The only option is to restart the computer.

====Shutdown via iLO====

Connecting the to the iLO of SCEVAL is quite simple.  The IP to be used within Putty is 10.148.64.233. If you are on a Mac, find a linux workstation (i.e., Winston) to connect as the security is too old for Mac to connect these days.

Once the connection has been opened, click into the terminal and press the enter key unit you get asked for a username:
    Username (case sensitive):    Administrator
    password:    ---

At the main menu, enter 'co' in order to switch to the console (CTRL+B always gets you back one menu layer if needed). 

Then hit the enter key again and provide username and password for the OpenVMS user 'system'. 

You then are connected to the workstation as system user and can shut it down properly (assuming there are no jobs running anymore on that node):
    $ shutdown
The progess of the shut down procedure will then be shown in the terminal. 

The following line will then indicate that the shutdown procedure has been completed:
    P00>>
To turn it off remotely (it does not automatically turn off after a shutdown), just press CTRL+B until you are back at the MP Main Menu. 

The enter 'cm'. To turn off the workstation, just enter the following command then:
    CM>    pc -nc -off
Turning back on the workstation works pretty much the same (with a slightly different command):
    CM> pc -nc -on

====Crash of XtremeCT II workstation computer====

For a non-responsive monitor and/or a mouse and/or a keyboard of the XtremeCT II workstation (sometimes caused by a lost of connection to server, or an internet outage):

:*1. The first thing to do is to check system logs for any indication of the cause of the issue, if you have the "Administrator" user privilege.
:*2. The next thing to do is to soft-reboot by remote-login and using command line to reboot the workstation, if you have the "system" user privilege.
:*3. If no success, try hard-reboot by pressing the power button as indicated below.
:*4. If no success, try hard-reboot by unplugging both power cords as indicated below.
:*5. If still no success, contact Scanco immediately.
:*6. The last thing to do is to record the issue and any possible reasons.

1.  Checking system logs:
 
:*1) Remotely log into the iLO (the management console of the workstation) using SSH as user "Administrator" with the system password. 
Both user/pw are case sensitive. For XtremeCT II workstation (James) it is SSH to 10.148.64.231. For others, consult the network diagram. 

A trick: This connection only succeeds through a MAC or PC (via putty) direct login, not through an OpenVMS or Scanco system.
   $ ssh -l Administrator 10.148.64.231
:*2) Then you get to a   MP>   prompt here:

   MP MAIN MENU:
         CO: Console
        VFP: Virtual Front Panel
         CM: Command Menu
         CL: Console Log
         SL: Show Event Logs
         HE: Main Help Menu
          X: Exit Connection
   [sc3401-mp]</> hpiLO->

:*3) Type in 'SL' to show event logs, then you could choose 'E' for system event:

   [sc3401-mp]</> hpiLO-> SL
     Event Log Viewer Menu:
       Log Name            Entries    % Full      Latest Timestamped Entry
  -------------------------------------------------------------------------------
   E - System Event          1661       58 %        13 Sep 2018 11:17:43
   F - Forward Progress     20000      100 %        13 Sep 2018 11:17:43
   I - iLO Event              500      100 %        18 Sep 2018 07:46:42
   C - Clear SEL and FPL
   L - Live Events

   Enter menu item or [Ctrl-B] to Quit: E
       Log Name            Entries    % Full      Latest Timestamped Entry
  -------------------------------------------------------------------------------
   E - System Event          1661       58 %        13 Sep 2018 11:17:43
     Event Log Navigation Help:
   +       View next block     (forward in time,  e.g. from 3 to 4)
   -       View previous block (backward in time, e.g. from 3 to 2)
   <CR>    Continue to the next or previous block
   D       Dump the entire log
   F       First entry
   L       Last entry
   J       Jump to entry number
   H       View mode configuration - Hex
   K       View mode configuration - Keyword
   T       View mode configuration - Text
   A       Alert Level Filter options
   U       Alert Level Unfiltered
   ?       Display this Help menu
   Q       Quit and return to the Event Log Viewer Menu
   Ctrl-B  Exit command, and return to the MP Main Menu

:*4) From here you can use -  and  +  to go through the log history back and forth. Not all logs do make a lot of sense though, as they are mainly meant for HPE engineers. 
:*5) When finished, use  CTRL + b to get back to the main MP> prompt.
:*6) Use  MP> co   to get to the console, where you see at what stage the booting process is, in case it does not boot properly.
:*7) When finished, use  CTRL + b to get back to the main MP> prompt.
:*8) Use  MP> x   to end the connection.

2. Soft-reboot by remote login:

If a workstation becomes unresponsive, it may be possible to shut it down remotely from another workstation or personal computer, which is preferable over doing a hard-reboot.  These instructions were provided by Mark Blanchard.

:*2.A) Remote shutdown from another micro-ct workstation.  MAY NOT WORK FOR CLUSTER SERVER

Login as the system user.
   $ set host 0
   user:  system
   password:

Start the system manager.
   $ mcr sysman

Set the environment to the workstation to be shutdown (e.g., thomas) using its node name (e.g., sc3309).
   SYSMAN>< set env/node=sc3309

Issue the shutdown command.  Add a time delay, e.g., 5 min, if you want to give users time to disconnect and the automatic reboot flag if you want to restart.
   SYSMAN> shutdown node /min=5 /automatic_reboot

You need to remain logged into the session running the shutdown until the other node has been shutdown completely!!!

:*2.B) Remote shutdown from a personal computer.
Begin a remote session to the workstation and login as the system user.

Create a detached terminal.
   $ create/terminal/detach
 
On the PC, port 6000 needs to be accessible to the micro-ct workstation and in the X-server's configuration (e.g., in file X0.hosts for Xming).  

Issue the shutdown command.
   $ shutdown

3. Hard-reboot by power button:

:*1) Hold the power button until it turns yellow.
:*2) Wait a minutes or two, then press the power button only once and shortly. You will not see anything for a while. Pressing the power button again during this time will turn off the workstation again !
:*3) After about 10~15 min, the workstation would boot up properly. 
  
4. Hard-reboot by unplugging both power cords:

:*1) Hard-reboot by unplugging both the power cords. This will reset the console and controllers.
:*2) Wait for 5~10 minutes, then press the power button only once and shortly. You will not see anything for a while. Pressing the power button again during this time will turn off the workstation again !
:*3) After about 10~15 min, the workstation would boot up properly.


[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]

====Crash of XtremeCT II workstation at Johnson Space Center====

Take a look at the log files by connecting to the iLO. Since we don't have an ethernet connection to the iLO at JSC, but we are able to log in on the LAN, we can log in as per normal and then connect:

 $ ssh "Administrator@192.68.196.57"

Use the <code>system</code> password to connect.

<pre>
hpiLO>    sl

                        e

                        t

                        a    3

                        d
</pre>

This then should create a dump in text form of all entries in the system log with an alarm level 3 (critical) and higher. Copy from the terminal and share with Scanco (e.g., Pascal).

[[Image:Redirect.jpg|15px]] [[#top|''Return to Contents'']] or [[Main_Page|''Return to Main Page'']]