Brickset parts CSV size?

Modeltrainman

Is there a reason my Brickset parts CSV is so big? Is there any way to reduce it? Bricklink allows importing of it to use as a Studio palate, but says mine is too big. @Huw? Is there a way to save my parts list as a Bricklnk XML?
?

Huw

No but that will most likely be too big too. How many parts are we talking?

Remove some sets from your collection to reduce it, that's the only way :)

Modeltrainman

Huw said:

No but that will most likely be too big too. How many parts are we talking?

Remove some sets from your collection to reduce it, that's the only way :)

My info says 370,340, right now. Thank you. It's kind of weird that they allow this, but I haven't found a way to shrink it. They say my Rebrickable one has errors, so won't accept it. I reached out to BL, but it's kind of strange that their integration with the best( in my opinion, at least) LEGO info site is easily broken.

Modeltrainman

I should add that sets imported fine. It's parts, which Studio wants, didn't. You'd think it could compare parts to what's listed in each set, but apparently not. So, I emailed them to see what they say.

davee123

I don't think I've ever noticed that export from BrickSet-- where is it?  Is it exporting distinct parts/colors?  What's the upper bound that it can handle? (I can easily imagine the file size for a BrickLink XML file being gigantic, considering the nature of the XML tags surrounding the data)

DaveE

Modeltrainman

Well, @davee123, all I did was go to the overview of my collection, under 'My Sets,', scroll down to parts, export the list of my parts, and BL says the CSV is too big for import. Must be under 1MB...

Huw

I don't think BrickLink import is designed to handle a third of a million parts!

@davee123, https://brickset.com/export.

davee123

Huh-- I'm guessing something's different about my collection (maybe too big?), since I'm not seeing "Parts" as a means of exporting?  I see "Sets" and "Minifigs", but not "Parts":



I think I can guess what the URL might be, but I'm not attempting it!

DaveE

oldtodd33

^ Huh, the parts should be left of your screen name. Maybe a setting? 

davee123

Indeed!  I didn't even know that setting existed!

DaveE

Huw

I don't advertise it much because if everyone used it it would generate Gb of data

davee123

So, getting back to the questions at hand:

"Is there a reason my Brickset parts CSV is so big?"

It looks like each line of the BrickSet CSV is roughly 94 characters or so per part on average.  I've got (for instance) 23,724 different parts that come back in the CSV (BrickSet thinks I have 856,859 parts), which means my CSV will be in the ballpark of  2,230,056 bytes (2.1M).

"Bricklink allows importing of it to use as a Studio palate, but says mine is too big."

Sounds like their limit is 1M, which (if my stats are correct) would mean you'd be looking at a ballpark of 11,155 distinct parts or less in order to fit a BrickSet CSV into a Studio palette.

"Is there a way to save my parts list as a Bricklnk XML?"

Yes! Teamwork with Rebrickable.

1st off, Rebrickable is probably the better tool for this anyway-- BrickSet's "My Parts" looks like it doesn't intermix inventories that it gets from Rebrickable, so it's missing parts that you might have from those sets (probably things that are older than the 1990s-ish ballpark)

If you have BrickSet export its set list with Rebrickable, you can then jump over there, and export as a BrickLink XML. (You'll need to link your Rebrickable account with your BrickSet account to do that)

However, the XML isn't all that much more efficient space-wise.  It looks like roughly 92-or-so characters on average for BrickLink XML format.  It's a little bit more efficient (because they don't have full descriptions), but the XML tags take up a lot of space.  So if they've got a similar 1M limit for BrickLink XML files, it might have the same problem.

Rebrickable seems to be a bit better, as it tries to resolve the disparities between its part identification system and BrickLink's, although I'm not sure how many discrepancies it has.  But it seems to do a pretty good job!

DaveE

Modeltrainman

So, Bricklink REALLY wants me to split my CSV, but any cut'npaste attempts have resulted in blank files.Going to keep looking.