Friday, November 8, 2013

Java HotSpot Options


Run Large Batch on Change Assistant



E1: CHA: Applying a Large Batch of ESUs, Such as Fix Current, Creates Many Pop-Up Errors: "Business function Library load failed - E:\JDEdwards\E900\PLANNER\bin32\CHRM.dll" (Doc ID 1353471.1)

We all might see this issue when applying big batch of ESU.  At this point there is no resolution from Oracle also.

I have done some changes in CHA start parameter and this has resolved. Here is a solution I tested –

Change Target parameter from. This is for 64 bit Windows Deployment Server

Right click Change Assistant - > Properties - > Target

C:\Windows\SysWOW64\javaws.exe -localfile "C:\Users\jdeservice\AppData\LocalLow\Sun\Java\Deployment\cache\xxxxxxxxxxxxxx"

Add –Xms and –Xmx parameter as shown below.  Note the localfile path should be kept same and only add Java memory parameters.

C:\Windows\SysWOW64\javaws.exe -Xms=256m -Xmx=2048m -localfile "C:\Users\jdeservice\AppData\LocalLow\Sun\Java\Deployment\cache\xxxxxxxxxxxx"

Test this and share your result on comment.


1 comment:

Friday, August 9, 2013

Start JDE Queue's

Start Queue manually



Go to system/bin32 on Enterprise Server

For Windows -
jdequeue.exe JDE jdepassword PD910 QB7333 UBE 5

For Unix -
runque.sh JDE jdepassword PD910 QB7333 UBE 5

where jdepassword is a password of JDE
PD910 is an environment defined in JDE.INI of Enterprise Server

Thursday, August 1, 2013

Oracle: Spool to Excel from SQLPlus

Spooling the output from SQLPlus to log file is often in unreadable format.

To Spool SQLPLus output to Excel (.xls) file, the trick is to turn On "MARKUP HTML"

set markup html on
set feedback off

spool c:\test.xls

select * from testdta.f0101
spool off

set markup html off

Thursday, July 11, 2013

Improve Performance of Server Manager


Improve Performance of Server Manager  -

1.
Stop Server Manager Console Service
Regedit


HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\SCFMngmtConsole1\Parameters\Start

Click on Params

-Xmx786m
-Xrs
-Djde.home=C:\jde_home\SCFMC
-jar
oc4j.jar

New Value
-Xms512m
-Xmx1536m
-Djde.home=C:\jde_home\SCFMC
-jar
oc4j.jar

Start Server Manager Console Service

2. Another way to change is look for document on oracle support 1082765.1






Tuesday, July 9, 2013

Web client not openning PDF


Clicek View PDF on Submitted Job and PDF not opening.

2 Solution to resolve :

Sol 1) Press Control Key quickly after your click View PDF

Sol 2) Internet Option - Security - Trusted Site - Custom level

Verify below setting are Enable.

Download - File Download - Enable
Download - Font Download - Enable

Wednesday, May 29, 2013

Weblogic 11g Start Stop Script on Linux

How to start Weblogic Service automatically on restart ?
How to restart Weblogic instance  on background ?

This are question that keeps fight our mind for weblogic on unix.  I have made a script to restart Weblogic instance automatically on restart. This will stop and start Weblogic and Server Manager Agent in sequence. You can use this script and change parameters to adjust your environment.

 The below script is for Linux system where executable is placed in /etc/init.d to start Weblogic 11g.


#
# This is to be saved on /etc/init.d
# Permissions chmod 755
# Owned by root:root
# Install using chkconfig --add weblogic
#
#
######################################
### Following parameter is set /etc/profile
# MW_HOME=/u01/app/wls1035/Middleware/

case $1 in
    start)
    su - oracle -c 'nohup /u01/jde_home/SCFHA/bin/startAgent > /dev/null &'
    su - oracle -c 'nohup $MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh > $MW_HOME/wlserver_10.3/common/nodemanager/nodemanager.out &'
    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/startWebLogic.sh > $MW_HOME/user_projects/domains/E1_Apps/servers/AdminServer/logs/AdminServer.out &'
    until nc -z jde91web 8000
      do
       sleep 10
      done

    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/startManagedWebLogic.sh NE1SVR t3://jde91web:8000 > $MW_HOME/user_projects/domains/E1_Apps/servers/NE1SVR/logs/NE1SVR.out &'
        until nc -z jde91web 8016
                do
                sleep 10
        done
        ;;

    stop)
    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/stopManagedWebLogic.sh NE1SVR t3://jde91web:8000 > $MW_HOME/user_projects/domains/"E1_Apps"/servers/NE1SVR/logs/NE1SVR.out &'
    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/stopWebLogic.sh > $MW_HOME/user_projects/domains/E1_Apps/servers/AdminServer/logs/AdminServer.out &'
    ps -ef | grep NodeManager | grep -v grep | awk '{system("kill -9 "$2)}'
    su - oracle -c 'nohup /u01/jde_home/SCFHA/bin/stopAgent > /dev/null &'
    ;;

    restart)
    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/stopManagedWebLogic.sh NE1SVR t3://jde91web:8000 > $MW_HOME/user_projects/domains/E1_Apps/servers/NE1SVR/logs/NE1SVR.out &'
    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/stopWebLogic.sh > $MW_HOME/user_projects/domains/E1_Apps/servers/AdminServer/logs/AdminServer.out &'
    ps -ef | grep NodeManager | grep -v grep | awk '{system("kill -9 "$2)}'
    su - oracle -c 'nohup /u01/jde_home/SCFHA/bin/stopAgent > /dev/null &'
    su - oracle -c 'nohup /u01/jde_home/SCFHA/bin/startAgent > /dev/null &'
    su - oracle -c 'nohup $MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh > $MW_HOME/wlserver_10.3/common/nodemanager/nodemanager.out &'
    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/startWeblogic.sh > $MW_HOME/user_projects/domains/E1_Apps/servers/AdminServer/logs/AdminServer.out &'
    until nc -z jde91web 8000
      do
       sleep 10
      done

    su - oracle -c 'nohup $MW_HOME/user_projects/domains/E1_Apps/bin/startManagedWebLogic.sh NE1SVR t3://jde91web:8000 > $MW_HOME/user_projects/domains/E1_Apps/servers/NE1SVR/logs/NE1SVR.out &'
        until nc -z jde91web 8016
                do
                sleep 10
        done
        ;;
esac

Friday, May 24, 2013

Identifying and Troubleshooting Call Object Kernel Zombie in JD Edwards (JDE)


In JD Edwards EnterpriseOne, a Call Object Kernel (CO Kernel) Zombie is a critical runtime issue where the kernel appears active but is stuck in a degraded or unstable state. It typically shows abnormal CPU/memory usage, thread stalls, and eventually leads to processing failures.

This guide explains how to identify a zombie kernel and trace the root cause using Server Manager, logs, and call stack analysis.




1. How to Identify a Call Object Kernel Zombie

Step 1: Check Kernel in Server Manager

  • Open Server Manager
  • Navigate to Call Object Kernel
  • Observe:
    • CPU usage
    • Memory consumption
    • Note baseline values

Step 2: Check JDB Connections and Cache (8.98.4+)

If you are on Enterprise Server 8.98.4 or above, monitor:

  • Open JDB connections
  • Cache count
    Record abnormal spikes.

Step 3: Analyze Threads

  • Go to kernel details
  • Review active threads
  • Ignore:
    • WRK:IDLE threads (not relevant)
  • Focus on active worker threads (WRK:*)

Look for:

  • Threads stuck or consuming resources abnormally
  • Non-idle worker threads

Step 4: Review jde.log

  • Open jde.log
  • Identify:
    • Last successful operations
    • First error or warning before kernel degradation
  • Look for the point where kernel behavior changes

Step 5: Analyze JDENET_N Logs

  • Open all JDENET_N logs
  • Identify:
    • Error messages linked to kernel PID
    • Any reference to zombie behavior or failure point

Step 6: Check Stack Reports

If available:

  • Review stack dump files
  • Identify:
    • Thread causing exception
    • Business Function involved

Step 7: Cross-Reference JAS Logs

  • Look for:
    • Call Object failures
    • “Associated kernel not found” errors
    • Unit-of-work failures

2. Example: Zombie Kernel Behavior Pattern

Observed Conditions

When kernel went into zombie state:

  • CPU usage: ~20% constant
  • Memory: 500–600 MB
  • Open JDB connections: ~30
  • JDE Cache count: 300–400
  • Total threads: 13
  • Idle threads: 9
       This is a Process Detail of Call Object Kernel went into Zombie. 

    

3. Identifying the Culprit Thread

Example thread analysis:

Thread IDType
3944SYS:Response Msg Listener
3396SYS:Dispatch (startup thread)
4724SYS:IPC SRV
3932WRK:SUPTON_055BD830_P4210
    

Conclusion

  • System threads are not the issue
  • Worker thread 3932 is the suspect
 

4. JAS Log Indicators

Example error:

*ERROR* COSE#1017 Associated kernel not found
Server: E1PRODENT3
PID: 1992
BSFN: F4211FSEditLine
User: SUPTON
Env: JPD900

Key observations:

  • Kernel PID: 1992
  • Business Function: F4211FSEditLine
  • User: SUPTON




Logs - 
Exception occurred in thread 3932. 
So 3932 is WRK:SUPTON_055BD830_P4210

This is likely a suspect but not always. There could be chance that some other thread made this culprit. So don't conclude, but make this as a suspect. 

User: SUPTON
Application: P4210

5. JDENET_N Call Stack Analysis

Call Stack (Top-to-Bottom Execution Flow)

Key execution chain:

F4211FSEditLine
 → CalculateSOMBFPriceCost
 → CalculateSalesPricesAndCosts
 → F4072CalculatePriceAdjustments
 → GetPrice
 → ConfigurationIDCacheFetch (CMFG.dll)

Critical Insight

  • The execution fails at:

    ConfigurationIDCacheFetch (CMFG.dll)
  • Cache behavior observed:
    • 300–400 entries during failure

 
 



Call Stack
CSALES.dll/UNKNOWN/F4211FSEditLine
CSALES.dll/UNKNOWN/CalculateSOMBFPriceCost
CDIST.dll/UNKNOWN/CalculateSalesPricesAndCosts
CDIST.dll/UNKNOWN/F4072CalculatePriceAdjustments
CCUSTOM.dll/UNKNOWN/GetPrice
CMFG.dll/UNKNOWN/ConfigurationIDCacheFetch

IN JDENET_N call stack is read Top to bottom
So the source of culprit if
CMFG,dll ConfigurationIDCacheFetch

Client Machine: USNEWSPDWL3
Look for log

23 May 2013 16:11:15,627[WARN][RUNTIME]*ERROR* CallObject@15d1951c: COSE#1017 Associated kernel not found. Please see Enterprise Server log for details: host E1PRODENT3:6015(49239) SocID:54906 PID:1992 BSFN:F4211FSEditLine user:SUPTON Env:JPD900
 
23 May 2013 16:11:15,642[WARN][RUNTIME]*ERROR* CallObject@15d1951c: Server problem. The server may still be available, but because of state information, the entire unit-of-work must be resubmitted user:SUPTON Env:JPD900 

Here from these logs
Server name : E1PRODENT3
PID: 1992
BSFN:F4211FSEditLine
User:SUPTON
Env:JPD900

Combining JAS.Log and JDENET.log this can be confimed
This Confirm F4211FSEditLine was a main BusinessFunction and Zombie occured because of CMFG.dll- ConfigurationIDCacheFetch

Note: In KRM we noted that the cache count when zombie occured was between 300-400


Now we go to Call Stack Thread - jde_1992_1369339853_1_dmp.log

 3932 -  WRK:SUPTON_055BD830_P4210


********************************************************
Thu May 23 16:10:53 2013
********************************************************
Generating call stacks for PID 1992 from PID 1992
********************************************************

====> Exception C0000005 ACCESS_VIOLATION occurred in thread 3932 with call stack:
0x7742e41b.! ntdll.dll 
RtlFreeHeap! ntdll.dll 
HeapFree! kernel32.dll 
free! MSVCR100.dll 
_jdeFreeInternal@4! jdel.dll 
_RBTREE_IndexDestroy@4! jdekrnl.dll 
_RBTREE_Destroy@8! jdekrnl.dll 
_JCACH_CacheDestroyInt@4! jdekrnl.dll 
_jdeCacheTerminate@8! jdekrnl.dll 
_ConfigurationIDCacheFetch@12! CMFG.dll 
_jdeCallObjectV2@44! jdekrnl.dll 
_jdeCallObject@40! jdekrnl.dll 
_GetPrice@12! CCUSTOM.dll 
_jdeCallObjectV2@44! jdekrnl.dll 
_jdeCallObject@40! jdekrnl.dll 
_I4500050_CallExtenalProgram@16! CDIST.dll 
_I4500050_CalculateAdjustment@16! CDIST.dll 
_F4072CalculatePriceAdjustments@12! CDIST.dll 
_jdeCallObjectV2@44! jdekrnl.dll 
_jdeCallObject@40! jdekrnl.dll 
_I4201500_CallAdvPricingAdjs@20! CDIST.dll 
_I4201500_CalculatePrices@16! CDIST.dll 
_CalculateSalesPricesAndCosts@12! CDIST.dll 
_jdeCallObjectV2@44! jdekrnl.dll 
_jdeCallObject@40! jdekrnl.dll 
_I4205120_CallCalcPricesAndCosts@76! CSALES.dll 
_I4205120_CalcPricesandCostsLineNotCancelled@80! CSALES.dll 
_CalculateSOMBFPriceCost@12! CSALES.dll 
_jdeCallObjectV2@44! jdekrnl.dll 
_jdeCallObject@40! jdekrnl.dll 
_I4200310_CalcPricesAndCosts@16! CSALES.dll 
_I4200310_F4211FSEditLine@12! CSALES.dll 
_jdeCallObjectV2@44! jdekrnl.dll 
_jdeCallObject@40! jdekrnl.dll 
_JDEK_ProcessCallRequest@24! jdekrnl.dll 
_JDEK_StartCallRequest@16! jdekrnl.dll 
_runBusinessFunction@4! jdekrnl.dll 
_runCallObjectJob@4! jdekrnl.dll 
_psthread_pool_job_execute@4! PSThreadUtils.dll 
?psthread_pool_worker_function@@YGPAXPAX@Z! PSThreadUtils.dll 
?threadFunctionWrapper@@YGPAXPAX@Z! psthread.dll 
0x761733aa.! kernel32.dll 
0x77439ef2.! ntdll.dll 
0x77439ec5.! ntdll.dll 

=====Call stack of thread 3932=====
_LogNTCallStackDump@12! jdel.dll 
?NTUnhandledExceptionHandler@NTExceptionHandler@@CGJPAU_EXCEPTION_POINTERS@@@Z! jdenet_k.exe 
0x761b003f.! kernel32.dll 
0x774774df.! ntdll.dll 
0x77439ec5.! ntdll.dll  

The Dump Call Stack is always read in thread from Bottom to Top.
Here is analysis

B4200310 - F4211FSEditLine@CSALES initiated a crash process
and cache occured _ConfigurationIDCacheFetch@12! CMFG.dll 

In between there are many other Business Function are called.  Note all these.

1. Keep Note of all information noted above
2. It is possible the CO Zombie was caused by other business function which no longer running at the point of crash
3. There was High memory and JDE Cache when crash occured.
4. Look for  SAR containing fixes for above pattern.

Result : F4211FSeditLine initiate Crash which occur on ConfigurationIDCacheFetch    

6. Root Cause Interpretation

From combined logs:

Primary Observations:

  • Kernel PID: 1992
  • Thread causing crash: 3932
  • Business Function: F4211FSEditLine
  • Failure point: ConfigurationIDCacheFetch
  • High cache usage at crash time
  • Memory corruption / access violation detected

7. Final Conclusion

  • The Call Object Kernel entered a zombie state due to instability in a worker thread

  • The most likely trigger:

    F4211FSEditLine → ConfigurationIDCacheFetch (CMFG.dll)

Key Findings:

  1. High cache usage (300–400 entries) contributed to instability
  2. Worker thread 3932 is the primary suspect
  3. Memory access violation occurred during cache cleanup
  4. Root cause may involve:
    • Cache corruption
    • Known bug in CMFG.dll logic
    • Or dependent Business Function failure

8. Important Notes

  • Do not immediately conclude root cause from one thread alone
  • Always correlate:
    • Server Manager metrics
    • JAS logs
    • JDENET logs
    • Stack dumps
  • Zombie kernels may be triggered by:
    • Earlier completed transactions
    • Detached memory references
    • Cache or DLL-level corruption

9. Recommendation

  • Search Oracle/SAR notes for:
    • ConfigurationIDCacheFetch
    • F4211FSEditLine
    • Cache-related memory leaks
  • Monitor:
    • JDE Cache thresholds
    • JDB connections
    • Kernel thread stability

Summary

A Call Object Kernel Zombie is not caused by a single symptom but by a combination of:

  • Thread failure
  • Cache corruption
  • Memory access violation
  • Business function execution chain failure

In this case, the execution flow leading to:

F4211FSEditLine → ConfigurationIDCacheFetch

is the strongest indicator of root cause.

1 comment:
Subscribe to: Posts (Atom)