Thursday, November 10, 2016

Automate startup of IBM Docs 2.0 server - the correct way?!

Hi.

I´ve done 5 different customer installations on IBM Docs 2.0 now, and they have all been having the same issues.
When I´ve done the installations, I´ve also created a scheduled restart of the server once a week. And after the server has restarted, the issues have been:
  • Viewer does not work as it should
  • Docs don´t work as it should
  • Thumbnail generation won´t work at all.
IBM initial setup, which is working fine, is that the scheduled tasks are set up to start on a "one time" trigger, and than with a 5 minute repeat interval.

Btw, do not alter this interval!! I tried... don´t do it.

And the scheduled tasks that are installed has a "when user is logged on" criteria.

When I have tried to change this to "wether or not user is logged on", and set the trigger to "on startup". in order to make the sched tasks start up automatically at windows reboot, without having to actually log on to the servers desktop... things was not working as expected.

The environments for the customers has been basically the same setup:
  • 1 IBM Connections server 
  • 1 Docs server where all three parts are installed (docs, viewer, conversion)
  • 1 DB2 server

My first installation was on a test server, where IBM Connections, DB2 and the IBM Docs servers were all installed on the same windows machine.
This server has no issues at all.

On the customer scenarios, where the Docs parts are installed on a separate server, a colleague of mine, @kilotin, created a startup script for the Docs parts. This script is executed through a Windows Scheduled task, and is triggered "on system startup".
The mounting of the IBM Connections shared folder is also something that´s called from within this script.

He also created a "stopdocs.bat" script, which shuts down the WAS server, node agents and disconnects the network drive mapping. It also does a reboot of the operating system.
(This script is shared later down in this blogentry, a modified version, in order to make this whole thing work.)

And, in order for the two Docs scheduled tasks ("sym_monitor" and "kill_timeout") to start automatically when the Windows server reboots, I´ve also modified those tasks to be triggered "on system startup" and to be executed with the option "start wether or not user is logged on".

All sounds good, right?
Well, not quite.

When I was in the process of developing this auto-start routine, this is what I did:
  1. Executed the "stopdocs.bat" script.
  2. Watched as the Websphere servers were shut down
  3. Then monitored that the OS was actually restarted.
  4. Logged into the desktop of the server, to monitor that the WAS processes were launched automatically.
  5. Then I tested each IBM Docs component manually (Create a document, edit, publish, print to pdf, uploaded documents, edited, watched as the thumbnail was generated.. etc, etc.)
Conclusion: Docs server is working fine and the restart routine works flawlessly also.

WRONG!

When a week went by, customers started complaining that the viewer didn´t work.
Especially documents of elderly Microsoft word format, ".DOC",  was not converting at all.
Newer document format, such as ".DOCX" was working....

SystemOut.log said stuff like:

[9/23/16 9:50:31:821 CEST] 0000047f QuerySnapshot E   CLFAF409W: Snapshot generation failed. Conversion error occurred. Error code:1203 Error message:Server returned unexpected status. Document id:ffbb67a7-5a98-49c8-8af7-28a74f86bc60 Document mimetype:application/msword Document version:1


And thumbnails previewing the content of the file is not shown either:

SystemOut.log said:
[11/8/16 19:51:46:095 CET] 00000132 UploadConvers I   CLFAF007I: Need to start a new conversion. This is for upload service. Doc id is 414ba791-1e60-4325-8e1d-d4f397b5014c. Mime type is application/vnd.openxmlformats-officedocument.wordprocessingml.document[11/8/16 19:51:46:188 CET] 00000132 ThumbnailServ I   Aquired thumbnail service lock successfully. DocId: 414ba791-1e60-4325-8e1d-d4f397b5014c LastModified: 1,478,631,105,495[11/8/16 19:51:46:188 CET] 00000132 ThumbnailServ I   CLFAF007I: Need to start a new conversion. This is for thumbnail service - one page upload conversion. Doc id is 414ba791-1e60-4325-8e1d-d4f397b5014c Mime type is application/vnd.openxmlformats-officedocument.wordprocessingml.document LastModified is 1478631105495[11/8/16 19:51:46:188 CET] 00000155 LCFilesEJBRep W   Can't find the file folder on EJB server! E:\shared\files\upload\files\35\65\eb4e2658-3cdb-4d5d-b79e-fe5637db363f[11/8/16 19:51:46:188 CET] 00000155 LCFilesEJBRep W   CLFAF703E: Get download document file. The doc id is 414ba791-1e60-4325-8e1d-d4f397b5014c[11/8/16 19:51:46:188 CET] 00000155 DocumentServi W   Image conversion for uploading failed. Document Id:414ba791-1e60-4325-8e1d-d4f397b5014c null


I then logged into the server, tested and then I see that ".DOC" documents started working again in the viewer. This was often after I did a manual restart of the Websphere Docs servers.
Thumbnails generation however, did not work.

Weird, right?

I then turned off the auto-restart scheduled task for each customer, while I was trying to figure this issue out. I made sure that the servers was running fine whilst doing so.

I tried everything I could think of. I created Windows services for the Docs WAS processes, ran those as an admin user, also tried as the SYSTEM user.... I tried mapping the network drive as different users, I was also in contact with IBM Champion Roberto Boccadoro, where we compared Docs configuration files with what he had in his system.... I then asked Roberto which user does he start the servers with. He was using "System". I then tried this on the scheduled task that starts up the servers as well.... Nothing gave results.

The thing that seemed to work, was actually ALWAYS logging manually into the Windows Desktop of the server.
That triggered stuff to work regarding .doc documents at least.

So, I then created a PMR, explaining what was going on. And after a few failed attempts, we were actually on to something!

IBM said that "you can´t use an Admin user in the Scheduled Tasks. You have to use the System user as the user that fires the Sched-tasks.". That means ALL THE TASKS, not only the Sched Task that starts the Websphere processes, but also the "sym_monitor" and "kill_timout" process.

And also, IBM also reccomends calling the Mount script in a separate script....... (which is wrong... I´ll discuss this later on).

Ok, first of all. When you install IBM Docs, the two scheduled tasks "sym_monitor" and "kill_timeout" is created with the user you are logged in as the installation was running. So, those two Sched tasks I had to manually set to run as "System".

I then restarted the Docs Windows server, and tested without logging into the desktop of the server this time.

Wow, now ".DOC" documents was actually working. Converted just fine and the viewer worked.

But one thing was still not working, and that was the thumbnail creation of the document.

I then stumbled upon an article (WHICH I DID NOT BOOKMARK and can´t find it again), describing the fact that when you map a drive in a script through a scheduled task, throug a "call" to a different mount script (as IBM suggested), and when this process is started up as the "SYSTEM" user, there is no sharing between the two processes.
Which means, the system user that starts up the WAS servers does not have access to the mapped drive, which was called through a separate mount script!!

So the solution, is to do the drive mounting inside the same script that starts up the Docs WAS processes. This is opposed to what IBM suggested.

So... This has been a long read for you guys. Here´s the fun part: How to correctly automate the startup of the Docs server and all it´s related processes:

First, you need to create this script:

(The order of starting up the WAS servers are important; 1. Conversion, 2. Docs, 3. Viewer.)


1. Create a directory e.g. D:\Scripts
2. Create a new subdirectory e.g. D:\Scripts\logs
3.Create file in D:\Scripts with the name "mapAndStartDocs.bat" with this content:
(Modify the script to suite your environment)
(I only map 1 drive, the Connections Shared Folder share, The docs_share and the viewer_share is locally on the Docs Windows server, which means I don´t need to map those to u: and v:)

@echo off


:Mount
echo Mapping IBM Connections Shared Data disk %date% %time% > D:\Scripts\logs\mount.log
net use E: \\appsrv1.skya.local\e$\IBM\Data /user:USERNAME PASSWORD /persistent:yes >> D:\Scripts\logs\mount.log

:StartNodes
echo Start conv nodeagent %date% %time% > D:\Scripts\logs\startNodes.log
call D:\IBM\WebSphere\AppServer\profiles\conv1Node01\bin\startNode.bat >> D:\Scripts\logs\startNodes.log

echo Start docs nodeagent %date% %time% >> D:\Scripts\logs\startNodes.log
call D:\IBM\WebSphere\AppServer\profiles\docs1Node01\bin\startNode.bat >> D:\Scripts\logs\startNodes.log

echo Start viewer nodeagent %date% %time% >> D:\Scripts\logs\startNodes.log
call D:\IBM\WebSphere\AppServer\profiles\viewer1Node01\bin\startNode.bat >> D:\Scripts\logs\startNodes.log

:StartServers

echo Start conv server %date% %time% > D:\Scripts\logs\startServers.log
call D:\IBM\WebSphere\AppServer\profiles\conv1Node01\bin\startServer.bat IBMConversionMember1 >> D:\Scripts\logs\startServers.log

echo Start docs server %date% %time% >> D:\Scripts\logs\startServers.log
call D:\IBM\WebSphere\AppServer\profiles\docs1Node01\bin\startServer.bat IBMDocsMember1 >> D:\Scripts\logs\startServers.log

echo Start viewer server %date% %time% >> D:\Scripts\logs\startServers.log
call D:\IBM\WebSphere\AppServer\profiles\viewer1Node01\bin\startServer.bat IBMViewerMember1 >> D:\Scripts\logs\startServers.log

:exit


Then create the script stopDocs.bat and put it in the same D:\Scripts folder.
stopDocs.bat content:

@echo off

:StopServers
echo Stop viewer server %date% %time% > D:\Scripts\logs\stopServers.log
call D:\IBM\WebSphere\AppServer\profiles\viewer1Node01\bin\stopServer.bat IBMViewerMember1 -username WASADMIN -password PASSWORD >> D:\Scripts\logs\stopServers.log

echo Stop docs server %date% %time% >> D:\Scripts\logs\stopServers.log
call D:\IBM\WebSphere\AppServer\profiles\docs1Node01\bin\stopServer.bat IBMDocsMember1 -username WASADMIN -password PASSWORD >> D:\Scripts\logs\stopServers.log

echo Stop conv server %date% %time% >> D:\Scripts\logs\stopServers.log
call D:\IBM\WebSphere\AppServer\profiles\conv1Node01\bin\stopServer.bat IBMConversionMember1 -username WASADMIN -password PASSWORD >> D:\Scripts\logs\stopServers.log

:StopNodes
echo Stop viewer nodeagent %date% %time% > D:\Scripts\logs\stopNodes.log
call D:\IBM\WebSphere\AppServer\profiles\viewer1Node01\bin\stopNode.bat -username WASADMIN -password PASSWORD >> D:\Scripts\logs\stopNodes.log

echo Stop docs nodeagent %date% %time% >> D:\Scripts\logs\stopNodes.log
call D:\IBM\WebSphere\AppServer\profiles\docs1Node01\bin\stopNode.bat -username WASADMIN -password PASSWORD >> D:\Scripts\logs\stopNodes.log

echo Stop conv nodeagent %date% %time% >> D:\Scripts\logs\stopNodes.log
call D:\IBM\WebSphere\AppServer\profiles\conv1Node01\bin\stopNode.bat -username WASADMIN -password PASSWORD >> D:\Scripts\logs\stopNodes.log

:UnmountShares
echo Unmount share %date% %time% > D:\Scripts\logs\unmount.log
net use * /delete /yes >> D:\Scripts\logs\unmount.log

:Reboot
echo Reboot %date% %time% > D:\Scripts\logs\reboot.log
shutdown -r -t 3

Then, change the two scheduled tasks like this:
kill_timeout:
Set the user to "SYSTEM", run with highest priv and "hidden"


Change trigger to "At startup" and 5 min repeat and duration to Indefinitely:


Do the same for "sym_monitor":




Then, create a new Scheduled task with the name "StartDocs".
Set the configuration as the following screenshots show:





Then, if you want to schedule a restart of the Docs server, create a Scheduled task with the name "StopDocs"






You can then test the restart out by right-clicking the "StopDocs" scheduled task and select "Run."

If all goes well, the WAS servers and NodeAgents will shut down, and a windows reboot will occur.
And when the server is back up again, you will be able to create Docs documents + upload files with .doc format and preview them + you will notice that the Thumbnail generation of the files are working as well.

Phew... This has been a huge headache... I´m thrilled that I´ve come up with the solution on how to automate the startup of the Docs servers with a scheduled reboot of the servers.

Cheers...


P.S. Any comments or questions or suggestions on improvements, please post them in the comments section.



Wednesday, November 9, 2016

IBM Sametime emoticons - free of charge :-)

Over the years, I´ve gathered up quite a few emoticons which I have in the Sametime Connect client.

Absolutely free of charge, from me to you :-)

How do you import them into the chat client?
Well, just go like this:

Open up a chat window, and click the smileyface in the toolbar.

Then, click "edit".

Then, to separate my emoticons from the standard ones that Sametime provides, click "new".


Enter a palette name:


Then click "import" and browse to the zip-file you´ve just downloaded.


And voila, the emoticons in the zip-file are all imported.


Then, go back to the chat window, click the smileyface and select your new palette in the dropdown list:


And remember, you can extend the size of this emoticon window, so that you can see more of the selectable icons:

Enjoy :-)



Wednesday, September 28, 2016

Docs 2.0 CR1 iFix 001 - Conversion app will not update.


Has anyone had issues with upgrading Docs CR1 to the CR1 iFix1?
This is the log error:

INFO This iFix is not for your current DocsConversion version. Please double check.

DocsApp and ViewerApp is updated fine, but not the conversion App.
I have gone through the applypatch.py file, and managed to figure out that the script looks for the file “conversion-config.json” in the IBMDocs-Config folder on the DMGR server. (D:\IBM\WebSphere\AppServer\profiles\Dmgr01\config\cells\appsrv1Cell01\IBMDocs-config)

It looks for the string “ifix_version” with a value >=6
This is a value that does not exist in the conversion-config.json file.

So I manually edited the conversion-config.json file on the dmgr server, to include the “ifix_version” part. I set the value to 6:
It´s located under the "build-info" section and the format is:
    "ifix_version": 7, 

After I did this, the applypatch procedure worked fine, and the conversion app was upgraded sucessfully.

I now see that the upgrade process has set the value to “7”.

"build-info": {    "build_version": "2.0.0.1",     "build_description": "IBM Connections Docs 2.0.0",     "ifix_version": 7,     "product_name": "IBM Connections Docs",     "build_timestamp": "20160824-1613"  },


The upgrade history of the server is: Docs 2.0. Then CR1 and then CR1 iFix 001.

I did not install Docs 2.0 ifix 001, 002, 003, 004, 005 first, since the CR1 package should include all of those.

Is this a bug in the upgrade process? Should the "ifix_version" string have been inserted into the conversion-config.json file in one of the previous Docs 2.0 iFix´es? 001? 002? 003?....
I don´t know. But this worked as a nice workaround.

Tuesday, September 6, 2016

Connections 5.5 Rich Text widget loops the community - a third option to fix the issue

DISCLAIMER WARNING:
I do not know if this is the correct way to fix the issue (changing the rteJAASAuth user). I have not verified this with IBM yet!!
--------------------------------------------------------


When doing a migration to 5.5 CR1, there´s an issue with the Rich Text widget.

Adding the widget in a community makes the browser go into a loop of refreshing the page.

Previously, this has helped me fix the issue (thanks you guys!):
https://brandlrainer.blogspot.no/2016/01/connections-55-rich-text-widget-in.html
and
https://www.urspringer.de/2016/06/15/ibm-connections-rich-content-widget-stops-working-migrating-server/

And there´s also a technote on this:
http://www-01.ibm.com/support/docview.wss?uid=swg21979899

In previous migrations, the above mentioned links has worked for me. (Deleting the "conn-rte" entry in OH2P_CLIENTCFG table, located in the HOMEPAGE database, deleting the message stores and switching to a LDAP user + set "everyone" as the reader of the Websphere Application named "RichTextEditors".)

But this time, I discovered another approach that allows me to still use the local "wasadmin" user, and not an LDAP user as the ConnectionsAdmin user.

There is a JAAS - J2C Authentication data called "rteJAASAuth" which I was not aware of.
This one was just laying there with no username and password set.

So I set it to match my ConnectionsAdmin user.

I then shut down the servers, deleted the messageStores, synced the nodes and deleted "temp" and "wstemp" and then started the servers up again.

After this, I had no browser "loopage" in the Community anymore. Problem fixed!

Is this a new role introduced in CR1? Why is it not set during installation of Connections?

In my next migration, I will attempt on setting this JAAS role only, and see if this is enough for the issue to be fixed. I might end up doing all the other stuff as well... we´ll see :-)


Tuesday, August 9, 2016

IBM Connections Docs 2.0 CR1 upgrade failure. Here´s the fix.

I attempted my first Docs 2.0 CR1 upgrade today, on a fresh Docs installation.

It failed.

I got the same error as Roberto Boccadoro did as described here:
https://rob59blog.wordpress.com/2016/07/29/ibm-connections-docs-cr1-installation-may-fail-with-an-error-on-conversion-server/




Roberto, thanks for this solution.
Although I ran into a new issue in step 4 of your solution.


I was not able to run the command:
upgrade_node.bat -installroot D:\IBM\ConnectionsDocs\Conversion -symcount 8

This is the error I got:

Traceback (most recent call last):  File "upgrade_node.py", line 5, in
    from config import CONFIG  File "D:\temp\docs_remote_installer\installer\config.py", line 46, in
    CONFIG=Config()  File "D:\temp\docs_remote_installer\installer\config.py", line 38, in __init__    self.get_input()  File "D:\temp\docs_remote_installer\installer\config.py", line 17, in get_input    if(not os.path.exists(options.install_root)):  File "d:\Python27\lib\genericpath.py", line 18, in exists    os.stat(path)TypeError: coercing to Unicode: need string or buffer, NoneType found

I then opened up the "upgrade_node.bat" file, and I noticed that the only thing it does is run python on the "upgrade_node.py" script.

After searching the net, I noticed a comment about the fact that python does not accept arguments unless they are defined with 2 hyphens, aka "--"

So I ran this command, which has 2 hyphens on the "installroot" and the "symcount" argument.

python upgrade_node.py --installroot d:\IBM\ConnectionsDocs\Conversion --symcount 8

This worked for me and Step 4 of Robertos guide worked.