Page 1 of 2 12 LastLast
Results 1 to 15 of 22

Thread: particles getBufData

  1. #1
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300

    particles getBufData

    I'm trying to read a particle buffer in one go but Lightwave seems to return a buffer that's bigger than particlecount * size:

    So for the position of 100.000 particles, I expect 1.200.000 bytes (100.000 X 3 X 4 bytes).
    But I need to allocate at least 1.200.017 bytes or my plugin crashes.
    For 200.000 particles I need to allocate at least 2.400.145 bytes.

    Can anyone see any logic in this?

    creacon

  2. #2
    LightWave Engineer Jarno's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    598
    Are those the particle counts returned by getPCount() ?
    The number of bytes copied will be the element byte size * number of elements per particle * number of particles. For the LWPSB_POS buffer that would indeed be 12 bytes per particle. The only issue I can think of is that the number of particles isn't what you are expecting.

    ---JvdL---

  3. #3
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    int particlecount = LW_Particlefuncs->getPCount(GetScene()->simobjectlist[0]->particles->psys);
    char *chardata = new char[particlecount * 12];
    LW_Particlefuncs->getBufData(GetScene()->simobjectlist[0]->particles->posbuffer, chardata);

    particlecount returns exactly 200.000 but I need to add 145 to the new statement for it to work. Don't know where the 145 comes from and why I need to add 17 if I only use 100.000 particles.

    I tried with a void* instead of the char* too, same problem.

    creacon

  4. #4
    TrueArt Support
    Join Date
    Feb 2003
    Location
    Poland
    Posts
    7,900
    Fill memory with well known code f.e. 0xc0de, then dump what was written to log file, and we will see difference..

  5. #5
    TrueArt Support
    Join Date
    Feb 2003
    Location
    Poland
    Posts
    7,900
    Sounds like you will have to add these additional bytes, whether you want or not, otherwise even if it's bug and will be fixed, you would have to make plugin requiring v11.6.1 or so to even start it..

  6. #6
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    As a temporary fix this will work, but I'll look into it a bit further, separating this problem from the rest of the code, to see what happens.
    I had problems with these buffers in the past using ->init to initialize a buffer and then using addparticle to add particles one by one on top of that buffer.
    When I used addparticle from the start (no init) to fill the buffer and addparticle again afterwards to add more particles everything works fine. I asked the question, reported this as a bug, but never got a solution.

    creacon

  7. #7
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    Jarno,

    I don't know if you have time to test this (or tell me what I am doing wrong), but this is what my tests show:
    the code below works, but if you comment out the "init" and uncomment the for loop (that should have the same effect), it crashes.

    // move this to your handler callback
    // LW_Particlefuncs = (LWPSysFuncs *)global( LWPSYSFUNCS_GLOBAL, GFUSE_TRANSIENT );

    LWPSysID testps = LW_Particlefuncs->create( LWPSB_POS ,LWPST_PRTCL );
    LWPSBufID posbuffer = NULL;

    // for (int i = 0; i < 200000; i++)
    // {
    // LW_Particlefuncs->addParticle(testps);
    // }
    LW_Particlefuncs->init(testps, 200000);

    posbuffer = LW_Particlefuncs->getBufID(testps, LWPSB_POS);
    int mycount = LW_Particlefuncs->getPCount(testps);
    char *mypartdata = new char[mycount * 12];

    LW_Particlefuncs->getBufData(posbuffer,mypartdata);

  8. #8
    LightWave Engineer Jarno's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    598
    Oh, wow, yes there is a bug. The getBufData() is using the amount of particles reserved in the buffer instead of the number in use. That can be a difference of up to 100 particles if there have been no particle removals, or more otherwise.

    It looks like this bug has been there forever. The same problem exists with setBufData().

    ---JvdL---

  9. #9
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    It's an easy fix then, any chance I can get my hands on a fix so I can give the crowd some liquids ;-)

    creacon

  10. #10
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    I implemented a naive workaround for the moment, getting one particle at a time and copying it to my buffer before caching to disk. It works but it is slow.

    I have no idea how these buffers are managed internally but there is lots of room for improvement, eg calling addparticle 3000000 times is dog slow. It's my guess there is some crude resizable array in there, that expects 10.000 particles instead of a few million.

    creacon

  11. #11
    LightWave Engineer Jarno's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    598
    The buffers are incremented by lots of 100 particles if the number of particles exceeds the current maximum. So yes, 3 million addParticle() calls will be slow as it has to resize the arrays about 30000 times. The init() function is meant to bulk create a large number of particles and will be much, much faster. But there is no function to add a large number of particles in one go to an existing system unfortunately.

    We are aware of the shortcomings of the current particle system. It was designed way back in the day for thousands of particles instead of the millions that is increasingly becoming the norm.

    I'm fixing this particular bug with the wrong buffer size being used. I don't know what patch or release it can go into yet.

    ---JvdL---

  12. #12
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    OK, thanks.
    I did a test with a c++ vector. add 3000000 structs with x,y,z values. I didn't even bother to use "reserve". time needed: none (it's so fast I can't press start and stop on my stopwatch, and I am too lazy to implement code that measures the time taken :-)

    Using addparticle:
    Stopped it after 10 minutes.

    I could easily use my own buffers, but then none of the other plugins would be able to get to the data (like HV and the instancing system).
    I'm going to track down what went wrong with the init followed by addparticles too.


    creacon

  13. #13
    TrueArt Support
    Join Date
    Feb 2003
    Location
    Poland
    Posts
    7,900
    addparticles, or any dynamic list implementation, has to reallocate array, copy old one to new memory, then add element at the end of list.
    So in the worstest scenario reallocation might happen even every single item.
    It's optimized by introducing chunks f.e. 16 items reallocation, then next one 32 items, 48 items and so on.
    You might see such field in dynamic list C++ classes (around the world) allowing to tell how many items reallocation will take place.

  14. #14
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    I know that , my point is that a c++ vector does that too, automatically and apparently fast too. I meant that in LW there is some C code that tries to obtain the same result but fails miserably. If I skip the LW particle buffers I could get the PhysX data in and out at least 2 times faster. Particle generation would be (according to my test) 100 times faster. Needless to say there is room for improvement.

    creacon


    Quote Originally Posted by Sensei View Post
    addparticles, or any dynamic list implementation, has to reallocate array, copy old one to new memory, then add element at the end of list.
    So in the worstest scenario reallocation might happen even every single item.
    It's optimized by introducing chunks f.e. 16 items reallocation, then next one 32 items, 48 items and so on.
    You might see such field in dynamic list C++ classes (around the world) allowing to tell how many items reallocation will take place.

  15. #15
    creacon
    Join Date
    Nov 2005
    Location
    Belgium
    Posts
    1,300
    Jarno,

    this code illustrates the problem with an ->init folowed by ->addparticle.
    The read loop reads about 2000000 particles then crashes (the countafter reports 5000000). In short: init works but the buffer doesn't get bigger when using addparticles after that.
    You'll see the difference if you comment out the init, the loop that creates the particles becomes very slow because the buffer is being expanded. If you leave the init in, the loop exits immediately.

    creacon
    Code:
    // 	move this to your handler callback
    // 	LW_Particlefuncs	=	(LWPSysFuncs *)global( LWPSYSFUNCS_GLOBAL, GFUSE_TRANSIENT );
    
      	LWPSysID testps =  LW_Particlefuncs->create( LWPSB_POS | LWPSB_ID ,LWPST_PRTCL );
      	LWPSBufID posbuffer = LW_Particlefuncs->getBufID(testps, LWPSB_POS);
      	int id;
      	float testpos[3];
      	testpos[0] = 1.0f;
      	testpos[1] = 2.0f;
      	testpos[2] = 3.0f;
    	LW_Particlefuncs->init(testps, 2000000);
    
      	for (int i = 0; i < 3000000; i++)
      	{
    	id =	LW_Particlefuncs->addParticle(testps);
      	LW_Particlefuncs->setParticle(posbuffer,i,testpos);
      	}
    
    	float readpos[3];
    
    	int countafter = LW_Particlefuncs->getPCount(testps);
    	for (int i = 0; i < countafter; i++)
    	{
    		LW_Particlefuncs->getParticle(posbuffer,i,readpos);
    	}

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •