JANOS Help System: [Commands] [Topics] [Tech Support] [Printable Manual] [Search]

Troubleshooting System Operation JANOS is designed with the philosophy that the tools needed to understand and diagnose system operation are an important part of the product itself. An embedded controller should not become a black box when something goes wrong. JANOS provides facilities for observing running processes, monitoring application output, examining communications, capturing network traffic, testing I/O, reviewing system activity and collecting diagnostic information. These tools are intended to help isolate a problem while the system is operating and, whenever practical, without disturbing that operation. Automation systems typically operate as a chain of cause and effect. A digital input may change state, or a command may arrive through a serial port or network connection. That stimulus originates with an external system and triggers built-in logic, a supplied application or a custom program. Decisions are made and actions follow, perhaps changing relay states or transmitting commands to another external system. When the expected result does not occur, the problem may exist anywhere along this chain. The cause may be external to the JNIOR, within its communications, logic or application processing, at its outputs, or in the external system receiving the resulting action. Effective troubleshooting means working along this chain to determine where expected behavior breaks down. JANOS provides diagnostic tools that help observe and verify each step, allowing the source of a problem to be isolated rather than assumed. These tools can be invaluable in locating the root cause even when the problem ultimately lies outside of the JNIOR. APPROACHING THE PROBLEM Begin by clearly defining the problem. What happened, and what should have happened instead? What event or condition should have triggered the expected action? Being able to state the problem in these terms provides a starting point for troubleshooting and is also valuable information should you need to contact technical support Consider what may have changed. Did the automation operate properly in the past, or is this a new function that has not yet worked as expected? Look for changes to the JNIOR, its configuration or applications, but also consider what may have changed elsewhere in the system. Equipment, wiring, network configuration, software and operating conditions outside of the JNIOR may all affect behavior. Next, identify the chain of events involved. Determine the source of the triggering input and how it reaches the JNIOR. Identify the logic or application that processes that input and makes the necessary decisions. Finally, determine what output or communication should result and where it goes. Each of these points can then be checked independently to determine where actual behavior first differs from what is expected. Timing may also be important. Note when the problem occurred and any external conditions that existed at the time. Consider whether the system was actively processing events or whether the problem appeared following a period of inactivity. Logs and other timestamped events can be particularly valuable in establishing the sequence of activity leading up to a problem and in correlating JNIOR activity with events elsewhere in the system. Make sure that the date and time on the JNIOR are correct. Much of the diagnostic information maintained by JANOS is timestamped, and those times may need to be correlated with logs or events from other equipment. Normally the JNIOR uses NTP to maintain accurate time. An incorrect clock can make otherwise useful diagnostic information difficult to correlate. LOOKING FOR SYSTEM CHANGES When an automation that previously operated correctly begins behaving differently, first look for evidence that something in the system has changed. JANOS maintains information that can help establish when configuration changes, errors and other significant events occurred. Checking this information before changing anything further may identify the cause directly or provide an important point in time from which to investigate. Syslog Events The system log, stored in the jniorsys.log file, records significant JANOS activity and should be reviewed for events near the time the problem first appeared. From the command line the following CAT command can be used to display recent log entries. cat -p jniorsys.log This content can also be viewed in the Syslog tab of the JNIOR WebUI. It may even be searched by using GREP, EDIT or MORE In particular, look for configuration changes such as modifications to Registry keys, as well as application, system and other events that might correspond with the change in behavior. Errors and Exceptions The presence of an errors.log file is also significant, but its timestamp may be as important as its contents. Determine when the file was last written and whether that time corresponds with the problem being investigated. If it does, or if it has been written even more recently, examine the file for exceptions, errors or other information that may identify what occurred. When a file is modified its Last Modification date is updated. This timestamp may be viewed by listing the folder content using the LS command as follows. If you omit the file name the entire folder contents are listed by descending modification date. ls -lv errors.log You may then use the CAT command or editor features as mentioned above to examine the errors.log file content. General Logging You might also look for any application specific LOG files that may have been updated around the time of the issue. Some applications keep additional information in separate LOG files. While logs are generally written to the root folder of the file system, there may be logs located in longer term storage. You can use the FIND command to look for those. For example: find .log There are additional techniques for monitoring applications. Those are typically not enabled by default, but can be activated. It becomes necessary to wait for the next occurrence of the issue. These will be discussed in a subsequent section. VERIFYING INPUT TRIGGERS An automation sequence may be triggered by a change in a digital or analog input, data received through a serial port, a command or message received over the network, or an internally generated event such as a timer or scheduled action. The trigger may originate directly from connected equipment or indirectly from another controller, computer, application or network service. The first step is to verify that the expected trigger actually reached the JNIOR. Confirming Digital Input Changes If an action is triggered by a change of input state and the event did not occur, then it is important to check if there was any input change. This can be done using the IOLOG command. For example: iolog -o The -O option here directs the output to the terminal for viewing. By default the command copies the log to the jniorio.log file. You may choose to do that and then examine the file content with CAT or the ED editor. The log details the state of inputs (DIN) and relay outputs (ROUT) and their changes down to the millisecond. Here is a sample: 06/30/26 13:05:21.552, DIN ---- 0000 0000, RLY ---- ---- 0000 0000 06/30/26 13:07:09.908, DIN ---- 0000 0001, RLY ---- ---- 0000 0000 06/30/26 13:12:38.101, DIN ---- 0000 0000, RLY ---- ---- 0000 0000 06/30/26 13:17:39.072, DIN ---- 0000 0001, RLY ---- ---- 0000 0000 06/30/26 13:29:01.186, DIN ---- 0000 0000, RLY ---- ---- 0000 0000 If the desired action did not occur, check the IOLOG to determine if the input state did change at that moment. If there was no I/O event then the problem is likely external to the JNIOR. Either the remote signal was not sent or wiring is no longer proper. If the input signal did change but there wasn't the desired response, you will need to look further down the chain. If the action is triggered by some other means and the response is dependent on input states, check to ensure that the inputs and the response are properly correlated. Did the right input states generate the right response? You may need to record the input status for use later in your diagnosis. This also shows the relay output states. If the result of an automation action is a change of relay status, you can see how the IOLOG command can provide confirmation of that as well. It is possible to artificially generate events for testing purposes. This can be a useful advanced debugging technique but should be approached with caution. The JNIOR cannot determine the consequences of an automation action. Simulating an input or forcing an output might activate equipment, trigger alarms or cause other unexpected actions in connected systems. Such testing should only be performed when the purpose of the I/O and the resulting automation actions are fully understood. Verifying Serial Triggers External equipment may communicate with the JNIOR using a Serial connection. This is typically RS-232. We highly recommend that this be achieved using the AUX serial port. This is located at the top of the JNIOR to the right of the power and Sensor Port connectors. The COM (labelled RS-232) port provides access to the Command Line Interface (CLI). During boot there are additional diagnostic messages transmitted through this port. Both of these could interfere with the protocol that the remote system is using. This could be the cause of some difficulty. The AUX port supports additional communications features that are not available on the COM port. One of those is the Transmission Log. This is accessible using the IOLOG command as follows. Here the -A option selects the AUX log and the -O option directs the output to the terminal screen. iolog -ao Characters (or bytes) passing through this port are logged. This command displays that traffic. An example follows. Here you can determine if the remote system is properly communicating and sending the correct messaging format. -- 10/02/26 14:45:06.879 646.790 72 75 6E 20 72 65 73 65 74 5F 73 68 6F 77 0D 0A run reset_show.. -31--30--2F--30--32--2F--32--36--20--31--34--3A--34--35--3A--30- 10/02/26 14:45:0 -37--2E--35--34--31--20--2B--20--65--78--65--63--75--74--69--6E- 7.541 + executin -67--5F--6D--61--63--72--6F--20--72--65--73--65--74--5F--73--68- g_macro reset_sh -6F--77--0D--0A--31--30--2F--30--32--2F--32--36--20--31--34--3A- ow..10/02/26 14: -34--35--3A--30--38--2E--35--37--36--20--2B--20--65--78--65--63- 45:08.576 + exec -75--74--65--64--5F--6D--61--63--72--6F--20--72--65--73--65--74- uted_macro reset -5F--73--68--6F--77--0D--0A- show.. -- 10/02/26 14:45:55.518 46.917 72 75 6E 20 65 78 74 72 61 5F 70 61 75 73 65 0D run extra_pause. 0A -31--30--2F--30--32--2F--32--36--20--31--34--3A--34--35--3A- .10/02/26 14:45: -35--36--2E--32--30--33--20--2B--20--65--78--65--63--75--74--69- 56.203 + executi -6E--67--5F--6D--61--63--72--6F--20--65--78--74--72--61--5F--70- ng_macro extra_p -61--75--73--65--0D--0A--31--30--2F--30--32--2F--32--36--20--31- ause..10/02/26 1 -34--3A--34--35--3A--35--37--2E--30--33--37--20--2B--20--65--78- 4:45:57.037 + ex -65--63--75--74--65--64--5F--6D--61--63--72--6F--20--65--78--74- ecuted_macro ext -72--61--5F--70--61--75--73--65--0D--0A- ra_pause.. If the outgoing characters display but those incoming are missing or unintelligible, then a serial port parameter mismatch might be the cause. For instance, both the JNIOR and the remote system must use the same Baud rates. That would be a configuration setting in both cases. Often a message must be precisely terminated with a CR ('\r' 0x0D), LF ('\n' 0x0A), or CRLF sequence of both. This is where you can see that detail and confirm the termination used. This is a common concern in Digital Cinema. Here we note that an automation action might also result in a message transmitted by the AUX port. This port can be used to trigger events either within the JNIOR or externally or both. The IOLOG command is therefore a useful diagnostic tool in either case. The AUX port can also be used in protocol conversion applications. Protocol conversion is, after all, a type of automation. Verifying Network Triggers Often these days, the Local Area Network (LAN) plays a key role in equipment communications. The JNIOR is a hardware I/O device with wired inputs and relay outputs, yet it is not uncommon for it to serve in a network facing role. Network messages can trigger events on the JNIOR and the JNIOR can drive other devices through the network. This can include protocol conversion. The LAN can be busy. The NETSTAT command displays the status of TCP/IP connections. This shows the active servers on the JNIOR side as well as any remote connections in progress. If the JNIOR is supposed to be connected to an external system pending a trigger, that connection should be displayed in this list. You can confirm it with NETSTAT. The NETSTAT -M monitoring mode shows the current status and updates that every couple of seconds. If an external device is supposed to connect to the JNIOR at a certain time, you can watch it happen. If the automation is to make an outward connection in response to a trigger, you can watch that happen as well. Any keystroke exits the monitoring. As an aside, the THD, CAT and PS commands also support monitoring modes. Both NETSTAT and PS allow the specification of up to 8 text files that can be displayed along with the network or process status updates. This feature performs the CAT function that lists additional lines added to the file in real time. You can get creative in observing connections, processes and logging simultaneously. If an event in the past did not go as expected, the NETSTAT -P command and related options can be used to display captured network traffic. JANOS supports a network sniffer similar in function to Wireshark. In fact, the NETSTAT -C command generates a PCAPNG file in the /temp folder. This can be downloaded. It will open up in Wireshark if you have that installed on your PC. Unlike Wireshark, JANOS continuously captures network traffic. The amount of time captured depends on the defined size of the capture buffer. You can set filters to reduce the capture traffic and to focus that on specific equipment. You can also use filters to select what traffic is displayed or saved then from the buffer into the PCAPNG file. This might allow you to confirm the communication that would have triggered an event even if it is considerably in the past. The NETSTAT -S command with associated options works just as the -P option. This, however, will display network traffic in real time. This allows you to watch events unfold. VERIFYING RESULTING ACTIONS Once the expected trigger has been confirmed, it is useful to next determine whether the intended result occurred. There is generally no need to investigate the internal processing if the JNIOR received the expected trigger and produced the expected response. In that case, troubleshooting can continue beyond the JNIOR with the external equipment or system that should have acted on that response. Many of the same diagnostic tools used to verify triggers can also confirm resulting actions. IOLOG can confirm relay output changes, the AUX transmission log can verify serial messages sent to external equipment, and network monitoring or packet capture can confirm communications transmitted over the network. DEBUGGING PROCESSING If you have confirmed that the expected trigger reaches the JNIOR and either there is no response, or the response is not that expected, the next step is to determine how the trigger should have been processed. This requires identifying the function or application responsible for processing the trigger and verifying that it is properly configured and operating as expected. Some actions can be accomplished using built-in capabilities. These would not require an active application program. An example would be the change of relay state in response to a JNIOR Protocol message. Both the JNIOR Protocol and the JANOS Management Protocol JMP are built-in functions that can control outputs. These protocols can also report input status to remote systems. So logically a protocol message might be sent in response to a trigger such as an input change. In either case the operations are configured using Registry settings. Searching Configuration While an application might store configuration information in an INI file or file with some other extension, most applications, especially those built-in functions, keep their configuration in the Registry. The Registry contains keys which are named entries. Each of those has a value. You can examine and alter Registry settings using the REGISTRY or REG command. To support debugging like this, JANOS logs all Registry changes in the jniorsys.log syslog file. You can search for recent Registry changes using the following command: + cat jniorsys.log* | egrep "Removed|Added|Changed" This command uses CAT to access the content of the jniorsys.log file. The '*' wildcard causes CAT to include the content of jniorsys.log.bak. We include that because the LOG ages to BAK after a time and we look back further this way. The '|' passes the output of that command to EGREP which selects lines containing "Removed", "Added", or "Changed". The uppercase characters are important although you can include the -I option for a case-independent search. Note that if your JNIOR runs JBAKUP, even older events will be stored in /flash/baks/jniorsys.log.zip. Assuming that the issue at hand occurred recently, looking back into ancient history may not be necessary. If you do need to look further back, and JBAKUP is active, you can use the following. cat /flash/baks/jniorsys.log/jniorsys.log.bak | egrep "Removed|Added|Changed" This uses the fact that, in some cases, a ZIP library may be accessed as a virtual folder. Here the CAT command reaches into that compressed library and produces the content of the likely large jniorsys.log.bak file stored in there. If you suspect that a Registry key may have been changed, you can access a last modification timestamp for the key. Here the REG command fetches that information. bruce_dev2 /> reg -m IO/Inputs/din1/Inversion [2605041245] IO/Inputs/din1/Inversion = disabled The timestamp here is in the format YYMMDDHHMM. On this bruce_dev2 JNIOR a search into the JBAKUP compressed library does reveal that change on that date. Unfortunately, most users rely on the main administrator account login. So it is not possible to say who or what made that change. Evidence of that might appear in some other LOG file. Relative to the issue at hand, look for configuration changes that might apply to the source of the event trigger, the resulting action, and the protocol (or application) that processes the trigger. For example, in a case using serial communications the Baud rate may have changed. For protocols you need to ensure that they are enabled and on the correct port. Both the JNIOR and the remote system must agree on these things. The entire Registry may be listed using the command: reg . You may include the -X option to display additional system keys that are defaulted. This output might be piped into GREP or EGREP as well. Alternatively, you may examine the /flash/jnior.ini file. This contains a backup of the Registry. This file is generated by the system and should not be externally modified. Make Registry changes using the REG command. There are many conditions that can, if changed, disrupt operation. Hopefully something that has changed will suggest the cause. Verifying Execution Status Some operations require a running application. For example, the MODBUS protocol is very popular with industrial controls. This protocol can be used to report input status or to change relay states. It is available for both client and server roles. MODBUS is supplied with the JNIOR. That application must be running. An application is generally started at boot using a Registry Run key. If the application doesn't start, a missing or improper Run key may be at fault. For a situation that was working and recently encountered an issue, a Run key change should be apparent when searching for configuration changes. Applications can properly start and then encounter a situation that might cause them to stop. In this case some error or exception would be logged. The PS command can be used to determine if an application is running. This can also tell you how the application was started. Note in the following that the MODBUS Server is running as well as other applications. Other system processes are indicated. bruce_dev2 /> ps 0: Idle Process 1: Network Service 2: System 3: Run/dblogs 4: Run/flash/modbusserver.jar 5: Command/Serial 6: Secure Transport 7 total 8.989 uptime 44.8K free pram This command was executed after this unit was up for only 9 seconds. Note that pram refers to internal Processor RAM. This is a limited resource and the 44.8K reports a normal amount. A more detailed (verbose) process status is available. bruce_dev2 /> ps -v runtime % mem msg hnd stk frm id desc 5:20.186 83 0: Idle Process 23.307 06 3023.1K 8 17 3 3: Run/dblogs 9.251 02 2.7K 8 27 2: System 3.073 00 489.1K 8 17 10 4: Run/flash/modbusserver.jar 1.244 00 16.1K 8 24 1: Network Service 0.117 00 64.3K 8 14 5: Command/Serial 6 total 6:21.290 uptime 77.9K free pram Here we see that 6 minutes later, one process has terminated, the amount of pram has increased, and MODBUS continues to run. Here we see how much CPU activity is attributed to each process at that moment. We also see the amount of standard memory that process has consumed. The other parameters are message, handle, and frame counts. The stk value is the largest percentage of process stack required by the process (highwater mark). Much larger memory usage on the order of MB, might indicate an application problem. High message, handle or frame counts might also be an indication of an issue. A highwater stack percentage over 80 should also be of concern. Processes may routinely utilize nearly 100% of the CPU. That is normal. But that condition must not persist indefinitely. In these cases it is best to contact the author of the application about the concern. You can start by contacting INTEG support When an Application Stops If an application should be running but does not appear to be, there should be some indication of what caused it to stop. Assuming that the Registry Run key looks proper and that the program (JAR) file is present in the /flash folder, the next step is to look for an errors.log file. Java applications encounter exceptions and these often provide a short message indicating the cause. This information will be sent to the errors.log file. Check for this file and its timestamp using the LS -L folder listing command. If the file is present, you may examine its content using CAT If you find something there that is not immediately obvious to you, please feel free to contact INTEG for assistance. GETTING BACK UP AND RUNNING Often there is not time to debug. Especially if that process is foreign to you. There is a tendency to REBOOT and hope for the best. In fact, that is often recommended by a lot of technical support people. If you need to get back up and running, there is no problem in rebooting. Just understand that there is a good chance of the problem occurring again. Before rebooting or making changes, consider using the JNIOR Support Tool to collect a snapshot of the system if circumstances permit. The snapshot preserves diagnostic information and system status that may be useful later in determining what occurred. This is particularly valuable when immediate recovery is more important than investigating the problem while it is present. We do not recommend, however, purposely establishing a reboot schedule. In an analogy with aviation, the high risk operations are takeoff and landing. Those are analogous to startup and shutdown. Both carry risk that operation will be disrupted and may not return to normal. JNIOR users tend to reboot the product by pulling power briefly. While we have worked hard to prevent it, abruptly removing power stops transactions in midstream. A partial memory write leaves the system in a corrupt state. This may not be corrected at boot and lead to additional problems. You should use the soft REBOOT command or perform the reboot through the Support Tool or other protocol. The JNIOR has been designed to run 24/7/365. In other words there is no real need to start fresh. Not with any frequency. PREPARING FOR RECURRENCE Restoring normal operation does not necessarily mean that the cause of a problem has been identified. If the problem is intermittent, it may be necessary to wait for it to occur again before enough information can be collected to determine the cause. JANOS provides several facilities that can be enabled or prepared in advance to preserve additional evidence when that happens. Some of these facilities monitor activity and create additional logging. Others establish a baseline that can later be used to determine what changed. The appropriate approach depends on what is known about the problem and which parts of the automation chain remain suspect. Capturing Application Output Sometimes we execute applications in the foreground from the command line. This would allow us to observe debugging information as it is displayed by the application. So applications have been developed to output such information. When they are run in the background, this output is normally discarded. An issue with this approach is that the application must be disturbed. It has to be stopped and then started in a command console session. That session then has to be monitored in order to catch the event as it occurs. This is not convenient. The disruption of ongoing execution could also reset the cause and not lead us to a solution. JANOS v2.6 now allows us to collect application output in the background as well. This can be enabled and disabled without disturbing the running application. This is particularly useful when investigating intermittent problems where restarting the application might alter or eliminate the condition being investigated. Applications may output information to the stdout stream. In a console session the stdout stream is brought to the display. In the background it is usually ignored. However, if the application sees an existing stdout.log file in the root directory, it will now append its output to that file. Normally the stdout.log file is not present. Output is discarded. We can create the file using the TOUCH command as follows: + touch stdout.log Now the file exists. It will appear in the directory listing. And it will grow as output from applications is collected. This is a LOG file and it will age to a BAK file just as others do. But now we can monitor it selectively from the command line using CAT as follows: cat -m stdout.log This will display the last few lines that exist in the file and display any new output as it occurs. Any keystroke interrupts this monitoring. The captured output in stdout.log can now be reviewed for clues surrounding the next occurrence of the issue at hand. Some applications also log this debug detail to their own LOG files. Sometimes those are stored in the /flash folder. Focusing on a Single Application Multiple applications can be executing on a JNIOR. Each could be providing debug output. All of it will be entangled in the stdout.log file. In some cases one application might overwhelm the others and mask needed debug output. Fortunately, you can select the output for a single application. To do so, you must name the LOG file using the program name. So for instance if the EKM.JAR program is running and we want only that debug output, you would create the file as shown here: touch stdout-ekm.log The program name is determined by what comes after the last '/' separator in the process name shown by the PS command. That name is placed after a dash in the file name as can be seen above. If both files exist, output from that application is directed to its application-specific file while output from other applications continues to be collected in stdout.log. To stop capturing the debug output, simply remove the file(s) with the RM command. You can also rename it with the REN command if you want to save content but stop capturing. Note that capturing diagnostic output consumes root folder file space. It should normally be disabled when it is no longer needed. Preserving Network History JANOS continuously captures network traffic in a ring buffer. When that buffer becomes full, the oldest captured packets are discarded to make room for new traffic. On a busy network this can significantly limit how far back you can look after an intermittent problem occurs. Capture filtering (see filter) can be used to extend that history. Generally it is better to exclude traffic known to be unimportant than to select only the traffic that you expect to need. Unexpected network activity may itself provide an important clue. For example, WebUI traffic can be excluded from the capture with a filter such as: netstat -f "!(80|443)" This can be particularly important when investigating a problem through the WebUI. Connecting to the JNIOR with a browser generates additional network traffic. Without appropriate filtering, the act of investigating the problem can consume capture space and discard some of the older traffic that you are trying to examine. The size of the network capture ring buffer can also be increased to retain additional history. Increasing the buffer and eliminating unnecessary traffic can substantially extend the period represented by the capture. The objective is to preserve enough network activity before, during and after the next occurrence to help establish what happened. Enlarging the Buffer The JNIOR by default allocates 512KB of memory for network capture. The NETSTAT -C command is used to generate a PCAPNG capture file which can be downloaded and opened with the Wireshark network protocol analyzer (!https://www.wireshark.org!wireshark.org). The default 512KB can represent minutes or even hours of network operation depending on the amount of network use. This packet buffer can be increased using this Registry key and can be set for any number of KB between 512 and 8192 (8MB Maximum). reg IpConfig/CaptureBuffer = 8192 An extremely large PCAPNG file can be difficult to upload and process. The NETSTAT -C command can also include a capture filter, writing only those packets of interest to the capture file. RECOMMENDATIONS If you encounter a problem, rebooting may get you back up and running. We suggest that you make note of the event and let INTEG know about it. We do not accept there being any level of background annoyance. While that might be normal for some other devices, it is unacceptable for JNIOR. To be proactive, please update your Series 4 JNIORs to the latest JANOS release. As of this writing JANOS v2.6 is current. We all avoid updates having been burnt by those at one time or another. A JANOS update should not be considered in that category of risk. These updates predominantly correct issues that have been identified. Those may not have affected you. But there is a chance that they actually have. And, you may not have noticed. Or, that they will in the future. Please install the update or run the Support Tool Update Project. If you are uncomfortable with that process, INTEG Support will be happy to walk you through it. SERIES 3 JNIOR As of this writing there are 25 year-old Series 3 JNIORs in operation. This product line has been a workhorse. The hardware does decay with time. The Flash memory components have a wear factor and will eventually fail. Batteries that retain LOG information and configuration go dead. Those cannot be easily changed on the Series 3. We cannot support the Series 3 at the level that we do with the current models. The debugging tools are just not there. Much of what you have read in this section of the manual pertains only to the Series 4 JNIOR. JANOS cannot run on the Series 3. In fact, the operating system those units run is completely unsupported by the original manufacturer. We can only work around that. We have pushed that approach to the limit. We recommend that you replace the Series 3 JNIOR with the corresponding Series 4. The Model 410 drops in for the Model 310 for example. All of the applications are available for the Series 4 that were available before. There is so much more performance and capability now. INTEG Support will be happy to work with you on the upgrade. We do not want you to struggle with older equipment. Especially when we are not in a position to support you as we would like.