[servicemix-user] Deadlock with bounded queue

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[servicemix-user] Deadlock with bounded queue

HURST, John B

Hi,

 

A couple of times I’ve experienced deadlocks with ServiceMix due to bounded queues becoming full. The most recent case was today, using ST flow. It’s easy to see how ST flow might be susceptible to this, but I believe I’ve had it with SEDA also. (Perhaps with a different queue/cause.)

 

In the case today I diagnosed it by examining the stack traces of threads using JMX (jconsole) after ServiceMix had hung. I found that seven threads are BLOCKED on synchronized (putGuard_) in BoundedLinkedQueue#put(), while one thread is stuck inside the method waiting for reconcilePutPermits() to return true.

 

My test case involves a FTPPoller receiving 1,100 files, passing them through to a XLST routing component, which passes them on to an HTTPInvoker component. (Actually both the FTPPoller and HTTPInvoker are local customizations and I’m using a couple of customized marshallers too.) I realise 1,100 files is quite a lot to receive at once via FTP. However, I increased the default capacity of BoundedLinkedQueue (from 1,024 to 10,240, as an experiment) and things seemed to work OK. I don’t really expect to have 1,100 files at once in real life, but in our scenario we will receive a couple hundred at once sometimes, so I thought this was a good test.

 

Should I raise a JIRA for this, or am I doing something stupid?

 

I have saved the stack traces.

 

BTW I’m using ServiceMix 2.0, built from source.

 

John Hurst

Wellington, New Zealand

 


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Reply | Threaded
Open this post in threaded view
|

Re: [servicemix-user] Deadlock with bounded queue

Robert Davies-2
Hi John,

please raise a jira issue - then we can all track progress on this etc. I'll have a look at this asap.

cheers,

Rob
On 9 Nov 2005, at 23:56, HURST, John B wrote:

Hi,

 

A couple of times I’ve experienced deadlocks with ServiceMix due to bounded queues becoming full. The most recent case was today, using ST flow. It’s easy to see how ST flow might be susceptible to this, but I believe I’ve had it with SEDA also. (Perhaps with a different queue/cause.)

 

In the case today I diagnosed it by examining the stack traces of threads using JMX (jconsole) after ServiceMix had hung. I found that seven threads are BLOCKED on synchronized (putGuard_) in BoundedLinkedQueue#put(), while one thread is stuck inside the method waiting for reconcilePutPermits() to return true.

 

My test case involves a FTPPoller receiving 1,100 files, passing them through to a XLST routing component, which passes them on to an HTTPInvoker component. (Actually both the FTPPoller and HTTPInvoker are local customizations and I’m using a couple of customized marshallers too.) I realise 1,100 files is quite a lot to receive at once via FTP. However, I increased the default capacity of BoundedLinkedQueue (from 1,024 to 10,240, as an experiment) and things seemed to work OK. I don’t really expect to have 1,100 files at once in real life, but in our scenario we will receive a couple hundred at once sometimes, so I thought this was a good test.

 

Should I raise a JIRA for this, or am I doing something stupid?

 

I have saved the stack traces.

 

BTW I’m using ServiceMix 2.0, built from source.

 

John Hurst

Wellington, New Zealand

 


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

Reply | Threaded
Open this post in threaded view
|

RE: [servicemix-user] Deadlock with bounded queue

John Hurst-6
OK,

I'll raise a JIRA when I get back to work in the morning and have the
details.  (Our hours are a bit different to most of you guys'.)

John Hurst

> -----Original Message-----
> From: Rob Davies [mailto:[hidden email]]
> Sent: Thursday, 10 November 2005 19:59
> To: [hidden email]
> Subject: Re: [servicemix-user] Deadlock with bounded queue
>
> Hi John,
>
> please raise a jira issue - then we can all track progress on this etc.
> I'll have a look at this asap.
>
> cheers,
>
> Rob
> On 9 Nov 2005, at 23:56, HURST, John B wrote:
>
>
>
>
> Hi,
>
>
>
> A couple of times I've experienced deadlocks with ServiceMix due to
> bounded queues becoming full. The most recent case was today, using ST
> flow. It's easy to see how ST flow might be susceptible to this, but I
> believe I've had it with SEDA also. (Perhaps with a different
> queue/cause.)
>
>
>
> In the case today I diagnosed it by examining the stack traces of
> threads using JMX (jconsole) after ServiceMix had hung. I found that seven
> threads are BLOCKED on synchronized (putGuard_) in
> BoundedLinkedQueue#put(), while one thread is stuck inside the method
> waiting for reconcilePutPermits() to return true.
>
>
>
> My test case involves a FTPPoller receiving 1,100 files, passing
> them through to a XLST routing component, which passes them on to an
> HTTPInvoker component. (Actually both the FTPPoller and HTTPInvoker are
> local customizations and I'm using a couple of customized marshallers
> too.) I realise 1,100 files is quite a lot to receive at once via FTP.
> However, I increased the default capacity of BoundedLinkedQueue (from
> 1,024 to 10,240, as an experiment) and things seemed to work OK. I don't
> really expect to have 1,100 files at once in real life, but in our
> scenario we will receive a couple hundred at once sometimes, so I thought
> this was a good test.
>
>
>
> Should I raise a JIRA for this, or am I doing something stupid?
>
>
>
> I have saved the stack traces.
>
>
>
> BTW I'm using ServiceMix 2.0, built from source.
>
>
>
> John Hurst
>
> Wellington, New Zealand
>
>
>
>
> ____________________________________________________________________
> __
> This email has been scanned by the MessageLabs Email Security
> System.
> For more information please visit http://www.messagelabs.com/email
> ____________________________________________________________________
> __
>
>


Reply | Threaded
Open this post in threaded view
|

Re: [servicemix-user] Deadlock with bounded queue

Robert Davies-2
John,

could you get a stack trace of this issue when it happens and attach  
it as well ?

cheers,

Rob


On 10 Nov 2005, at 07:57, John Hurst wrote:

> OK,
>
> I'll raise a JIRA when I get back to work in the morning and have the
> details.  (Our hours are a bit different to most of you guys'.)
>
> John Hurst
>
>> -----Original Message-----
>> From: Rob Davies [mailto:[hidden email]]
>> Sent: Thursday, 10 November 2005 19:59
>> To: [hidden email]
>> Subject: Re: [servicemix-user] Deadlock with bounded queue
>>
>> Hi John,
>>
>> please raise a jira issue - then we can all track progress on this  
>> etc.
>> I'll have a look at this asap.
>>
>> cheers,
>>
>> Rob
>> On 9 Nov 2005, at 23:56, HURST, John B wrote:
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>> A couple of times I've experienced deadlocks with ServiceMix due to
>> bounded queues becoming full. The most recent case was today,  
>> using ST
>> flow. It's easy to see how ST flow might be susceptible to this,  
>> but I
>> believe I've had it with SEDA also. (Perhaps with a different
>> queue/cause.)
>>
>>
>>
>> In the case today I diagnosed it by examining the stack traces of
>> threads using JMX (jconsole) after ServiceMix had hung. I found  
>> that seven
>> threads are BLOCKED on synchronized (putGuard_) in
>> BoundedLinkedQueue#put(), while one thread is stuck inside the method
>> waiting for reconcilePutPermits() to return true.
>>
>>
>>
>> My test case involves a FTPPoller receiving 1,100 files, passing
>> them through to a XLST routing component, which passes them on to an
>> HTTPInvoker component. (Actually both the FTPPoller and  
>> HTTPInvoker are
>> local customizations and I'm using a couple of customized marshallers
>> too.) I realise 1,100 files is quite a lot to receive at once via  
>> FTP.
>> However, I increased the default capacity of BoundedLinkedQueue (from
>> 1,024 to 10,240, as an experiment) and things seemed to work OK. I  
>> don't
>> really expect to have 1,100 files at once in real life, but in our
>> scenario we will receive a couple hundred at once sometimes, so I  
>> thought
>> this was a good test.
>>
>>
>>
>> Should I raise a JIRA for this, or am I doing something stupid?
>>
>>
>>
>> I have saved the stack traces.
>>
>>
>>
>> BTW I'm using ServiceMix 2.0, built from source.
>>
>>
>>
>> John Hurst
>>
>> Wellington, New Zealand
>>
>>
>>
>>
>> ____________________________________________________________________
>> __
>> This email has been scanned by the MessageLabs Email Security
>> System.
>> For more information please visit http://www.messagelabs.com/email
>> ____________________________________________________________________
>> __
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

RE: [servicemix-user] Deadlock with bounded queue

HURST, John B
In reply to this post by HURST, John B
Hi,

I haven't been able to get this problem to occur with a straightforward
reproducible setup.  So I'm starting to suspect something's wrong with
my component programming, which somehow is filling up a queue when it
shouldn't.

If I ever do get a reproducible scenario I will file a JIRA.

John Hurst
Wellington, New Zealand

-----Original Message-----
From: Rob Davies [mailto:[hidden email]]
Sent: Friday, 11 November 2005 02:03
To: [hidden email]
Subject: Re: [servicemix-user] Deadlock with bounded queue

John,

could you get a stack trace of this issue when it happens and attach  
it as well ?

cheers,

Rob


On 10 Nov 2005, at 07:57, John Hurst wrote:

> OK,
>
> I'll raise a JIRA when I get back to work in the morning and have the
> details.  (Our hours are a bit different to most of you guys'.)
>
> John Hurst
>
>> -----Original Message-----
>> From: Rob Davies [mailto:[hidden email]]
>> Sent: Thursday, 10 November 2005 19:59
>> To: [hidden email]
>> Subject: Re: [servicemix-user] Deadlock with bounded queue
>>
>> Hi John,
>>
>> please raise a jira issue - then we can all track progress on this  
>> etc.
>> I'll have a look at this asap.
>>
>> cheers,
>>
>> Rob
>> On 9 Nov 2005, at 23:56, HURST, John B wrote:
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>> A couple of times I've experienced deadlocks with ServiceMix due
to

>> bounded queues becoming full. The most recent case was today,  
>> using ST
>> flow. It's easy to see how ST flow might be susceptible to this,  
>> but I
>> believe I've had it with SEDA also. (Perhaps with a different
>> queue/cause.)
>>
>>
>>
>> In the case today I diagnosed it by examining the stack traces
of

>> threads using JMX (jconsole) after ServiceMix had hung. I found  
>> that seven
>> threads are BLOCKED on synchronized (putGuard_) in
>> BoundedLinkedQueue#put(), while one thread is stuck inside the method
>> waiting for reconcilePutPermits() to return true.
>>
>>
>>
>> My test case involves a FTPPoller receiving 1,100 files, passing
>> them through to a XLST routing component, which passes them on to an
>> HTTPInvoker component. (Actually both the FTPPoller and  
>> HTTPInvoker are
>> local customizations and I'm using a couple of customized marshallers
>> too.) I realise 1,100 files is quite a lot to receive at once via  
>> FTP.
>> However, I increased the default capacity of BoundedLinkedQueue (from
>> 1,024 to 10,240, as an experiment) and things seemed to work OK. I  
>> don't
>> really expect to have 1,100 files at once in real life, but in our
>> scenario we will receive a couple hundred at once sometimes, so I  
>> thought
>> this was a good test.
>>
>>
>>
>> Should I raise a JIRA for this, or am I doing something stupid?
>>
>>
>>
>> I have saved the stack traces.
>>
>>
>>
>> BTW I'm using ServiceMix 2.0, built from source.
>>
>>
>>
>> John Hurst
>>
>> Wellington, New Zealand
>>
>>
>>
>>
>>
____________________________________________________________________
>> __
>> This email has been scanned by the MessageLabs Email Security
>> System.
>> For more information please visit
http://www.messagelabs.com/email
>>
____________________________________________________________________
>> __
>>
>>
>


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________
Reply | Threaded
Open this post in threaded view
|

Re: [servicemix-user] Deadlock with bounded queue

Robert Davies-2
ok - thank John,  - I hate it when that happens :)

On 11 Nov 2005, at 02:01, HURST, John B wrote:

> Hi,
>
> I haven't been able to get this problem to occur with a  
> straightforward
> reproducible setup.  So I'm starting to suspect something's wrong with
> my component programming, which somehow is filling up a queue when it
> shouldn't.
>
> If I ever do get a reproducible scenario I will file a JIRA.
>
> John Hurst
> Wellington, New Zealand
>
> -----Original Message-----
> From: Rob Davies [mailto:[hidden email]]
> Sent: Friday, 11 November 2005 02:03
> To: [hidden email]
> Subject: Re: [servicemix-user] Deadlock with bounded queue
>
> John,
>
> could you get a stack trace of this issue when it happens and attach
> it as well ?
>
> cheers,
>
> Rob
>
>
> On 10 Nov 2005, at 07:57, John Hurst wrote:
>
>> OK,
>>
>> I'll raise a JIRA when I get back to work in the morning and have the
>> details.  (Our hours are a bit different to most of you guys'.)
>>
>> John Hurst
>>
>>> -----Original Message-----
>>> From: Rob Davies [mailto:[hidden email]]
>>> Sent: Thursday, 10 November 2005 19:59
>>> To: [hidden email]
>>> Subject: Re: [servicemix-user] Deadlock with bounded queue
>>>
>>> Hi John,
>>>
>>> please raise a jira issue - then we can all track progress on this
>>> etc.
>>> I'll have a look at this asap.
>>>
>>> cheers,
>>>
>>> Rob
>>> On 9 Nov 2005, at 23:56, HURST, John B wrote:
>>>
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> A couple of times I've experienced deadlocks with ServiceMix due
> to
>>> bounded queues becoming full. The most recent case was today,
>>> using ST
>>> flow. It's easy to see how ST flow might be susceptible to this,
>>> but I
>>> believe I've had it with SEDA also. (Perhaps with a different
>>> queue/cause.)
>>>
>>>
>>>
>>> In the case today I diagnosed it by examining the stack traces
> of
>>> threads using JMX (jconsole) after ServiceMix had hung. I found
>>> that seven
>>> threads are BLOCKED on synchronized (putGuard_) in
>>> BoundedLinkedQueue#put(), while one thread is stuck inside the  
>>> method
>>> waiting for reconcilePutPermits() to return true.
>>>
>>>
>>>
>>> My test case involves a FTPPoller receiving 1,100 files, passing
>>> them through to a XLST routing component, which passes them on to an
>>> HTTPInvoker component. (Actually both the FTPPoller and
>>> HTTPInvoker are
>>> local customizations and I'm using a couple of customized  
>>> marshallers
>>> too.) I realise 1,100 files is quite a lot to receive at once via
>>> FTP.
>>> However, I increased the default capacity of BoundedLinkedQueue  
>>> (from
>>> 1,024 to 10,240, as an experiment) and things seemed to work OK. I
>>> don't
>>> really expect to have 1,100 files at once in real life, but in our
>>> scenario we will receive a couple hundred at once sometimes, so I
>>> thought
>>> this was a good test.
>>>
>>>
>>>
>>> Should I raise a JIRA for this, or am I doing something stupid?
>>>
>>>
>>>
>>> I have saved the stack traces.
>>>
>>>
>>>
>>> BTW I'm using ServiceMix 2.0, built from source.
>>>
>>>
>>>
>>> John Hurst
>>>
>>> Wellington, New Zealand
>>>
>>>
>>>
>>>
>>>
> ____________________________________________________________________
>>> __
>>> This email has been scanned by the MessageLabs Email Security
>>> System.
>>> For more information please visit
> http://www.messagelabs.com/email
>>>
> ____________________________________________________________________
>>> __
>>>
>>>
>>
>
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________