We are seeing the following exception, leading the server to become unresponsive.
ERROR
11:46:16,829 ERROR [org.jboss.as.ejb3.invocation] (default-threads – 9) WFLYEJB0034: EJB Invocation failed on component mycomp_MySampleCMTEJB for method public abstract void aaa.bbb.ccc.MySampleCMTEJBLocal.methodOne(aaa.bbb.ccc..myVarOne,aaa.bbb.ccc.myVarTwo): javax.ejb.EJBException: javax.ejb.EJBTransactionRolledbackException: WFLYEJB0378: Failed to acquire a permit within 5 MINUTES
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:188) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
Solution
- The exception you have cited indicates exhaustion in the EJB’s
slsb-strict-max-pool
. You need to increase the number of instances in theslsb-strict-max-pool
to cater to the demand using themax-pool-size
attribute. Please increase theslsb-strict-max-pool
to 100 or 200 and observe the outcome. You need to define an idealmax-pool-size
based on the load of the EAP server. Here’s a sample configuration snippet for your reference:
<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" max-pool-size="200" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
....
....
</bean-instance-pools>
Root Cause
- The EJB container creates a new bean instance each time a bean is accessed until the container reached the configured max pool size. Then, the container would block incoming instance requests up to 5 minutes, and wait for the bean instances to be returned to the pooled by the clients. If the clients take longer than 5 minutes to return the bean instances to the pool, the incoming bean requests cannot be accommodated by the EJB container and throw the reported exception. The defaults are adequate for most deployments. However, you need to increase the instance pool size should the clients occupy bean instances for more than a few minutes.
Description:
Users were unable to connect to application with an error message “JBAS014516: Failed to acquire a permit WITHIN 5 MINUTES”We had the following configurations in place.
1. As per the product vendor recommendation, we have increased the max-pool-size from 20 (default) to 100 but still the error had occur.
2. Though we have increased to 100, it might not be sufficient for our user base, to derive this value vendor is suggesting us to monitor the EJB connection pool through JConsole but unfortunately we could not set up this monitor.
3. we have set the JVM memory to 4 GB but the same is not reflecting in task manager.
4. We have modified this parameter as follows
rem # Reduce the RMI GCs to once per hour for Sun JVMs.
set «JAVA_OPTS=%JAVA_OPTS% -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv6Stack=false»
Moreover there additional configurations being suggest by Vendor & Red Hat,
1. Modify module.xml file to define a dependency that allows the JBOSS connector module to use JMX Mbeans — Vednor
2. EJBs can use different pool configurations to allow finer control of the total number of each EJB type, however to find the suitable value for your environment, we would recommend testing. The default EJB pool can be disabled and EJBs will not use a pool, instead a new EJB instance will be created when a given thread needs to invoke a method on an EJB. — Red Hat
Questions:
1. How to set up the JConsole to monitor a JBoss’s (running as services instead of as a process via standalone.bat ) EJB connection pool? OR Is there any other free tools to monitor?
2. What are the parameters to be monitored in EJB connection pool?
3. Will it have an importance to reflect the JVM memory size in the task manager as 4 GB? If so, where to look in?
4. As per the documentation, JBoss JVM to prefer IPv4 but we have not enabled it. What will be the impact?
5. Any idea on the Vendor recommendation on module.xml file configuration changes?
6. As per the Red Hat, if we remove the bean-instance-pool-ref from the ejb3 subsystem, what will be the impact?
I hope, i have made myself clear if not please let me know if you have any other queries.
It is really hard to suggest a good approach without understanding of your use case, but most probably you are invoking a method on you EJB bean, that takes too long to execute, gradually exhausting the instances in pool until there is none left for the caller process. As more and more requests for this operation come in, the EJB container will try to provide the client with next free item in the pool. Normally, if the operation on bean instance finishes, the instance would be returned to the pool and could be used for next client call. If the operation takes long time, the pool will get exhausted until there is no available instance left to serve the client call. Based on your config, the EJB container has 20 instances; if none is available, it will try to wait for 5 minutes whether some instance wont be returned to the pool. If it fails to acquire an instance in that time, it will throw the above mentioned error to the caller.
So where does that lead us:
First and foremost analyze the EJB operation that takes that long(it is very useful to add a simple EJB interceptor to your deployment, that would track start and finish of you EJB calls plus track execution time)
Determine who calls that EJB instance — maybe it is doing excessive amounts of invocations against that bean.
If the long running operation can not be avoided or optimized, increase the pool size, so that more instances of that bean are available to the clients(adjust max-pool-size
)
If your use case requires long running operations, but does not need to block and wait for their result, consider asynchronous processing along with JMS queue — create jobs in the queue and then execute them using MDB. You can still store and query the status of your processing via DB.
- Unsolved Questions
- This Question
Eric-∑
We are getting an exception when calling a Salesforce SOAP web service from Java. We wrote a SOAP web service in Salesforce and call it from Java. It has been working for long time, but recently, we got an error when calling the web service from Java. The following are the details of the error. Does anybody know how to solve such issue? Is it an error thrown from Salesforce or Java specific issue? It will be appreciated if anybody can provide some suggestions. Thanks.
javax.xml.soap.SOAPException: Failed to get connectionid. Reason: Failed to get connection id. Exception: JBAS014516: Failed to acquire a permit within 5 MINUTES
at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1742)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2900)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
at org.apache.axis.Message.getSOAPEnvelope(Message.java:435)
at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)
at org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)
at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
at org.apache.axis.client.Call.invoke(Call.java:2767)
at org.apache.axis.client.Call.invoke(Call.java:2443)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)
at labvantage.labvantage.services.SapphireWS.SapphireWSSoapBindingStub.getConnectionId(SapphireWSSoapBindingStub.java:1499)
at com.appliedproteomics.labvantage.samplestatus.Sample.getSamples(Sample.java:237)
at com.appliedproteomics.labvantage.samplestatus.Main.fetchSamples(Main.java:111)
at com.appliedproteomics.labvantage.samplestatus.Main.main(Main.java:39)
You need to sign in to do that.
Dismiss
-
horaceho
- Influencer
- Posts: 22
- Liked: 1 time
- Joined: Nov 27, 2017 3:24 am
- Full Name: Horace
- Contact:
Failed re-install windows Agent
Hi all,
I had a windows agent need to migrate the agent from Veeam Server to another new Veeam Server,
after I uninstall the Veeam agent and re-install that it’s failed to install with error «Failed to acquire agent managed by (the old VBR Server)» —
and
“Veeam Agent for Microsoft Windows cannot be configured, because it is managed by your system administrator.» —
-
HannesK
- Veeam Software
- Posts: 12385
- Liked: 2380 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Failed re-install windows Agent
Post
by HannesK » Jun 25, 2019 6:05 am
Hello,
are you using the central deployment option from VBR or are you installing locally on the machine and then manually connect to a Veeam repository?
Did you also uninstall the Veeam installer service?
Best regards,
Hannes
-
horaceho
- Influencer
- Posts: 22
- Liked: 1 time
- Joined: Nov 27, 2017 3:24 am
- Full Name: Horace
- Contact:
Re: Failed re-install windows Agent
Post
by horaceho » Jun 25, 2019 7:14 am
Hi ,
2 of this I had tried , the first error message was show while I using deploy option and the 2nd message is from install locally .
Also I had uninstall the installer services and restarted the server.
-
HannesK
- Veeam Software
- Posts: 12385
- Liked: 2380 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Failed re-install windows Agent
Post
by HannesK » Jun 25, 2019 7:45 am
Hello,
I cannot reproduce it. It works fine for me.
I suggest to contact support. Please post the case number here for reference.
I did the following:
— VBR1: «uninstall all components» from one of the servers in one of my protection groups
— VBR2: create new job with individual computer & run backup job
— VBR2: «uninstall all components»
— VBR1: install agent again
no issues
I also tested to add the agent on VBR1 while it was managed by VBR2. This was not possible, but this is correct and expected.
Best regards,
Hannes
-
horaceho
- Influencer
- Posts: 22
- Liked: 1 time
- Joined: Nov 27, 2017 3:24 am
- Full Name: Horace
- Contact:
Re: Failed re-install windows Agent
Post
by horaceho » Jun 25, 2019 7:58 am
Hi,
I don’t know , there are a 3 node cluster servers and I had migrate the a and d node was success but it’s failed for the C node.
Support advise me for re install to the old VBR but it’s still failed after tried .
Case # 03628214
-
HannesK
- Veeam Software
- Posts: 12385
- Liked: 2380 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Failed re-install windows Agent
Post
by HannesK » Jun 25, 2019 9:41 am
I just reviewed the case and I have nothing to add. There must be some leftovers on the C node for whatever reason (I don’t understand which registry was checked already by you / support). Support is the only one who can help in that situation.
-
HannesK
- Veeam Software
- Posts: 12385
- Liked: 2380 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Failed re-install windows Agent
Post
by HannesK » Jun 25, 2019 3:34 pm
Hello,
yes, by clicking «uninstall» — but that failed obviously. Please escalate the case in the support portal if things are not going forward.
Best regards,
Hannes
-
horaceho
- Influencer
- Posts: 22
- Liked: 1 time
- Joined: Nov 27, 2017 3:24 am
- Full Name: Horace
- Contact:
-
horaceho
- Influencer
- Posts: 22
- Liked: 1 time
- Joined: Nov 27, 2017 3:24 am
- Full Name: Horace
- Contact:
Re: Failed re-install windows Agent
Post
by horaceho » Jul 04, 2019 2:24 am
after support troubleshoots, it’s solved by install cert to the agent client and re-install to the old VBR server and uninstall.
-
jessie0
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: Oct 16, 2019 5:04 am
- Contact:
Re: Failed re-install windows Agent
Post
by jessie0 » Dec 09, 2019 3:28 am
Hello Hannesk,
My question is slightly different from the above asked question(s), and sorry if this is not the right forum for these questions.
During my first attempt at backing up a physical server(windows OS),
I had to remotely install veeam agent from the veeam console into the server after the backup job failed.
Now I am experimenting again with the same OS and the backup job was successful, even though I did not install veeam agent into the server.
I confirmed from the «Inventory» option on the veeam console that Agent had been installed because it only gave me the «uninstall all component» option.
Questions:
1. Is veeam agent for windows OS installed automatically when a backup job is started? I am certain I did not install any agent.
2. Is there some option in the backup job configuration steps that is enabled by default to install veeam agent into physical server.
If yes, please point me in that direction.
Thanks.
-
jessie0
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: Oct 16, 2019 5:04 am
- Contact:
Re: Failed re-install windows Agent
Post
by jessie0 » Dec 10, 2019 5:23 am
Hello Wishr,
Thanks for your response.
You mentioned «Protection Group». The confusing thing is that I have not created a protection group with the server in it.
I don’t understand how the outcome is different for both attempts.
If I were to explain this to a customer, I would want to be sure of the correct procedure.
I cant say I understand the reason for the difference in both attempts.
Thanks.
-
wishr
- Expert
- Posts: 3077
- Liked: 451 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Failed re-install windows Agent
Post
by wishr » Dec 10, 2019 9:58 am
Hi Jessie,
Another possibility could be the Agent remains here after the initial installation during the first attempt. It’s hard to say what happened without looking at what you have done so I suggest you go the whole process one more time using a clean machine with no Agents installed.
Thanks
-
chalkynz
- Influencer
- Posts: 21
- Liked: 3 times
- Joined: Aug 06, 2019 2:02 am
- Full Name: Nathan Shaw
- Contact:
Re: Failed re-install windows Agent
Post
by chalkynz » May 15, 2020 9:39 pm
HannesK wrote: ↑Jun 25, 2019 7:45 am
…I also tested to add the agent on VBR1 while it was managed by VBR2. This was not possible, but this is correct and expected.
…
Hi, so this implies that the procedure for moving to a new Management server, if an old Management server is unavailable, is to uninstall all agents manually. Is that right?
-
wishr
- Expert
- Posts: 3077
- Liked: 451 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Failed re-install windows Agent
Post
by wishr » May 18, 2020 11:07 am
Hi Nathan,
Not necessarily. There is a registry key that allows a «new» backup server to get the ownership of the agents initially deployed from an «old» backup server. The key is obtainable from our support team.
Thanks
-
mradr
- Lurker
- Posts: 2
- Liked: never
- Joined: Jun 27, 2021 1:14 am
- Contact:
[MERGED] Trying to get Free to work on my computer again
Post
by mradr » Jun 27, 2021 1:22 am
Trying to get Free to work on my computer again. For some reason a few years back my copy just flat out stop working giving me the following error: «Veeam Agent for MS Windows cannot be configured because it is managed by your system administrator»
I must have tried 1000 different methods across the board, even waited for a few version differences to upgrade to to see if it would be fix over time. Still no luck so asking here…
Tried removing the veeam reg entries, removing left over folders in programs/files and programdata, SQL 2012, etc
Using the newest build 5.0 something and at this point figure I ask here. I don’t own a full copy ~ so not sure if I could ask for help on their support. Can’t open the GUI either. Yea .. just at a stuck point. Event logs don’t show anything error from what I can tell and even the logs Veeam makes says the same thing as well. I guess is there a «cleaner» way to reset the Veeam for a reinstall or what am I missing here? I didnt have it on any managed software either. This was just a stand alone computer back when Veeam 2.0 release at the time (well been using it from the beta release).
-
Natalia Lupacheva
- Expert
- Posts: 1143
- Liked: 302 times
- Joined: Apr 27, 2020 12:46 pm
- Full Name: Natalia Lupacheva
- Contact:
Re: Failed re-install windows Agent
Post
by Natalia Lupacheva » Jun 27, 2021 5:13 pm
Hi Travis,
Moved your post to the existing thread with a similar issue.
Please take a look at the discussions above.
If none of the suggested solutions help — please open a Support case and share its ID with us.
Thanks!
-
Mildur
- Veeam Software
- Posts: 4825
- Liked: 1441 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed re-install windows Agent
Post
by Mildur » Jun 27, 2021 5:49 pm
wishr wrote: ↑May 18, 2020 11:07 am
There is a registry key that allows a «new» backup server to get the ownership of the agents initially deployed from an «old» backup server. The key is obtainable from our support team.
I need to get this registry key too. We have decided to rebuild the mgmt backup server from scratch as a vm for our new backup environment. I have to reconfigure ownership for around 30 Agents to the new vbr server.
Would it be possible to write a KB article for this key or is there some risk to release it? But I‘m happy to open a support case for the key, if it‘s not possible
Product Management Analyst @ Veeam Software
-
mradr
- Lurker
- Posts: 2
- Liked: never
- Joined: Jun 27, 2021 1:14 am
- Contact:
Re: Failed re-install windows Agent
Post
by mradr » Jun 28, 2021 6:20 am
Natalia Lupacheva wrote: ↑Jun 27, 2021 5:13 pm
If none of the suggested solutions help — please open a Support case and share its ID with us.
The thing is I can’t create the ticket number. For some reason it wants me to «upgrade» my account but the email needs to be business and when I do it says the account is already in use xD so unable to create such thing because of the limit of the website logic. I do use it for business… so that account is lock out of use for personal.
Yes, I read over this post, did everything I knew how from here. Still no luck. Its like its reading from some other location, but I am not aware of all locations veeams puts files or reg keys. Something veeam doesnt seem to document anywhere short of just finding it by random on other websites. To be clear, this copy was never put on a VBR server or any managed method.
I went as far as to:
1) Uninstall the software
2) Uninstall SQL 12
3) Remove everything in programdataveeam
4) Remove everything in program filesveeam
5) Remove LocalDB (witch wasn’t created)
6) Check system32configsystemprofile
7) systemprofileappdatalocalms sql server local dbinstancesveeamendpoint
Remove veeam reg key in HKEY_LOCAL_MACHINESOFTWARE
Is there any other place Veeam hides stuff xD?
-
Dima P.
- Product Manager
- Posts: 13709
- Liked: 1381 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Failed re-install windows Agent
Post
by Dima P. » Jun 30, 2021 3:32 pm
1 person likes this post
Fabian, This is the one.
mradr, please PM me your contact email address — I’ll ask support team to get in touch with you. Thanks!
-
Mildur
- Veeam Software
- Posts: 4825
- Liked: 1441 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Failed re-install windows Agent
Post
by Mildur » Jun 30, 2021 3:56 pm
Thanks Dima
Product Management Analyst @ Veeam Software
-
anderp
- Lurker
- Posts: 1
- Liked: 2 times
- Joined: Oct 14, 2021 12:24 pm
- Full Name: Paul
- Contact:
Re: Failed re-install windows Agent
Post
by anderp » Oct 14, 2021 12:46 pm
2 people like this post
I didn’t want to do a full reinstall, so I went to the server that would not update and stopped «Veeam agent for Microsoft Windows» it failed to stop so I disabled it. I then restarted «Veeam installer service». Then switch to my Veeam backup server and ran update from the Inventory tab on the Veeam server. It finally installed the update to Windows agent. I went back to the server, switched «Veeam agent for Microsoft Windows» back to Automatic and started the services. My SQL transactions logs started backing up again. All updates completed last night.
Who is online
Users browsing this forum: No registered users and 9 guests
13/09/2022
Reading Time: 1–2 minutes
If you try to configure agent-based VM backups with agents managed by a different Veeam Veeam Backup & Replication server, you may experience the «Failed to acquire agent managed by…» error.
This error occurs when you perform the Rescan operation of Protection Groups containing Windows or Linux machines managed by agents centrally deployed from the Veeam Backup & Replication server.
If the processed machines were previously managed by a different Veeam Backup & Replication server, you may face the error:
Failed to acquire agent managed by VBR server.
Fix «Failed to acquire agent managed by…» error
There are two ways to fix the «Failed to acquire agent managed by…» error:
- Remove the Veeam’s agent
- Add a new key in the VBR registry
The deletion of installed agents can be time consuming if the number of machines to process is quite high and generally is not the ideal solution.
Modify the Windows Registry
An easy and quick way to prevent this error is the creation of a new registry key.
From your Veeam Backup & Replication server, open the Registry Editor and go to HKEY_LOCAL_MACHINESOFTWAREVeeamVeeam Backup and Replication.
Right click Veeam Backup and Replication and select New > DWORD (32-bit) Value option.
Name the create new key as AgentDiscoveryIgnoreOwnership and enter 1 as Value data. Click OK to confirm.
The new AgentDiscoveryIgnoreOwnership registry key has been created successfully.
Now go to the Inventory area and select Physical Infrastructure section. Right click the Protection Group to process and select Rescan.
The rescan operation for Windows machines is now completed successfully.
Also the rescan operation for Linux machines completes successfully.
With this simple configuration it is now possible to complete the rescan procedure without problems.
About The Author
Paolo Valsecchi
System Engineer, VCP-DCV, VCP-DM, vExpert, VMCE, VMCA, Veeam Vanguard. Working experience focused on VMware vSphere, Microsoft Active Directory, and backup/DR solutions.
In this brief article I’ll show some potential fix for warning “Nothing to process” that might happen when you’re trying to backup physical or virtual server.
Step to reproduce
When you have configure backup job and start this job you get “Nothing to process” warning:
Backup job stop immediate as result.
Issue
I have some server that have installed Veeam agent already:
But when I start rescan of manually added servers in “Inventory” tab I’ve found that this server cannot be processed correctly:
I always get “Details: Failed to acquire agent managed by VBR server.” warning:
Manually added server not displaying on “Manually added” list as result and cannot be backed up correctly.
Solution
On problem server previously was installed Veeam B&R agent for 9.5 version. Now I use 11 version of Veeam B&R. Agent was updated. But my guess is that some sort of configuration leave from old Veeam agent in OS.
I’ve found knowled base article that state that you can re-create the configuration database for Veeam Agent.
I just do follow things:
1. Remove all components of Veeam B&R agent.
2. Delete HKLMSOFTWAREVeeam registry location:
3. Reinstall Veeam B&R agent.
4. Perform rescan of manually added servers in “Inventory” tab.
Server was discovered correctly and backup performs correctly now.
We are seeing the following exception, leading the server to become unresponsive.
ERROR
11:46:16,829 ERROR [org.jboss.as.ejb3.invocation] (default-threads – 9) WFLYEJB0034: EJB Invocation failed on component mycomp_MySampleCMTEJB for method public abstract void aaa.bbb.ccc.MySampleCMTEJBLocal.methodOne(aaa.bbb.ccc..myVarOne,aaa.bbb.ccc.myVarTwo): javax.ejb.EJBException: javax.ejb.EJBTransactionRolledbackException: WFLYEJB0378: Failed to acquire a permit within 5 MINUTES
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:188) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [wildfly-ejb3-7.1.4.GA-redhat-1.jar:7.1.4.GA-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
Solution
- The exception you have cited indicates exhaustion in the EJB’s
slsb-strict-max-pool
. You need to increase the number of instances in theslsb-strict-max-pool
to cater to the demand using themax-pool-size
attribute. Please increase theslsb-strict-max-pool
to 100 or 200 and observe the outcome. You need to define an idealmax-pool-size
based on the load of the EAP server. Here’s a sample configuration snippet for your reference:
<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" max-pool-size="200" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
....
....
</bean-instance-pools>
Root Cause
- The EJB container creates a new bean instance each time a bean is accessed until the container reached the configured max pool size. Then, the container would block incoming instance requests up to 5 minutes, and wait for the bean instances to be returned to the pooled by the clients. If the clients take longer than 5 minutes to return the bean instances to the pool, the incoming bean requests cannot be accommodated by the EJB container and throw the reported exception. The defaults are adequate for most deployments. However, you need to increase the instance pool size should the clients occupy bean instances for more than a few minutes.
Loading
I am currently facing ‘JBAS014516: Failed to acquire a permit within 5 MINUTES’ issue with my EJB-JBOSS configuration.Below is my configuration —
<subsystem xmlns="urn:jboss:domain:ejb3:1.4">
<session-bean>
<stateless>
<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
</stateless>
<stateful default-access-timeout="5000" cache-ref="simple"/>
<singleton default-access-timeout="5000"/>
</session-bean>
<mdb>
<resource-adapter-ref resource-adapter-name="hornetq-ra"/>
<bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
</mdb>
<pools>
<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
</bean-instance-pools>
</pools>
<caches>
<cache name="simple" aliases="NoPassivationCache"/>
<cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
<cache name="clustered" passivation-store-ref="abc" aliases="StatefulTreeCache"/>
</caches>
<async thread-pool-name="default"/>
<timer-service thread-pool-name="default">
<data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
</timer-service>
<remote connector-ref="remoting-connector" thread-pool-name="default"/>
<thread-pools>
<thread-pool name="default">
<max-threads count="10"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
<iiop enable-by-default="false" use-qualified-name="false"/>
<default-security-domain value="other"/>
<default-missing-method-permissions-deny-access value="true"/>
</subsystem>
To resolve this should i increase the ‘strict-max-pool’ to a higher value or increase thread pool size.
asked Jun 2, 2017 at 5:42
Narendra PandeyNarendra Pandey
5143 gold badges11 silver badges26 bronze badges
1
It is really hard to suggest a good approach without understanding of your use case, but most probably you are invoking a method on you EJB bean, that takes too long to execute, gradually exhausting the instances in pool until there is none left for the caller process. As more and more requests for this operation come in, the EJB container will try to provide the client with next free item in the pool. Normally, if the operation on bean instance finishes, the instance would be returned to the pool and could be used for next client call. If the operation takes long time, the pool will get exhausted until there is no available instance left to serve the client call. Based on your config, the EJB container has 20 instances; if none is available, it will try to wait for 5 minutes whether some instance wont be returned to the pool. If it fails to acquire an instance in that time, it will throw the above mentioned error to the caller.
So where does that lead us:
First and foremost analyze the EJB operation that takes that long(it is very useful to add a simple EJB interceptor to your deployment, that would track start and finish of you EJB calls plus track execution time)
Determine who calls that EJB instance — maybe it is doing excessive amounts of invocations against that bean.
If the long running operation can not be avoided or optimized, increase the pool size, so that more instances of that bean are available to the clients(adjust max-pool-size
)
If your use case requires long running operations, but does not need to block and wait for their result, consider asynchronous processing along with JMS queue — create jobs in the queue and then execute them using MDB. You can still store and query the status of your processing via DB.
answered Jun 2, 2017 at 8:03
Description:
Users were unable to connect to application with an error message “JBAS014516: Failed to acquire a permit WITHIN 5 MINUTES”We had the following configurations in place.
1. As per the product vendor recommendation, we have increased the max-pool-size from 20 (default) to 100 but still the error had occur.
2. Though we have increased to 100, it might not be sufficient for our user base, to derive this value vendor is suggesting us to monitor the EJB connection pool through JConsole but unfortunately we could not set up this monitor.
3. we have set the JVM memory to 4 GB but the same is not reflecting in task manager.
4. We have modified this parameter as follows
rem # Reduce the RMI GCs to once per hour for Sun JVMs.
set «JAVA_OPTS=%JAVA_OPTS% -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv6Stack=false»
Moreover there additional configurations being suggest by Vendor & Red Hat,
1. Modify module.xml file to define a dependency that allows the JBOSS connector module to use JMX Mbeans — Vednor
2. EJBs can use different pool configurations to allow finer control of the total number of each EJB type, however to find the suitable value for your environment, we would recommend testing. The default EJB pool can be disabled and EJBs will not use a pool, instead a new EJB instance will be created when a given thread needs to invoke a method on an EJB. — Red Hat
Questions:
1. How to set up the JConsole to monitor a JBoss’s (running as services instead of as a process via standalone.bat ) EJB connection pool? OR Is there any other free tools to monitor?
2. What are the parameters to be monitored in EJB connection pool?
3. Will it have an importance to reflect the JVM memory size in the task manager as 4 GB? If so, where to look in?
4. As per the documentation, JBoss JVM to prefer IPv4 but we have not enabled it. What will be the impact?
5. Any idea on the Vendor recommendation on module.xml file configuration changes?
6. As per the Red Hat, if we remove the bean-instance-pool-ref from the ejb3 subsystem, what will be the impact?
I hope, i have made myself clear if not please let me know if you have any other queries.