Brickset ForumHome›
Shopping at LEGO or Amazon?
Please use our links: LEGO.com • Amazon
As an Amazon Associate we earn from qualifying purchases.
Please use our links: LEGO.com • Amazon
As an Amazon Associate we earn from qualifying purchases.
Automated BrickLink cost minimization program available
Shopping at LEGO.com or Amazon?
Please use our links: LEGO.com • Amazon
Recent discussions • Categories • Privacy Policy • Brickset.com
Howdy, Stranger!
Categories
- All Categories
- 16.2K General
- 3 Announcements
- 3.1K Collecting
- 3.7K Everything else LEGO
- 1.7K Building and Techniques
- 2.3K Brickset
- 14 Ambassador
- 1K Introduce yourself
- 385 The Database
- 438 Brickset.com
- 315 Forum Operation
- 15 FAQs
- 13.9K Shopping
- 1.7K Shopping USA/Canada
- 337 Shopping Elsewhere
- 11K Marketplace
- 1.2K Other
- 571 Community and Events
- 262 Photography/Video
Brickset.com is a participant in the Amazon Services LLC Associates Program, the Amazon.com.ca, Inc. Associates Program and the Amazon EU Associates Programme, which are affiliate advertising programs designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.
As an Amazon Associate we earn from qualifying purchases.
Comments
http://maques.hu/lego/auto-pab/pablist-us.bsx
All item matched to a BrickLink item, hopefully to the same item, but miracles can happen anytime...
BrickStore loads the file without error, but I had to change the ID of the otherwise proper BrickLink item "Tire 21mm D. x 6mm City Motorcycle" from ID 50861 to its former ID x1436 (changed May 27 this year)- as BrickStore was complaining about it. Now with x1436 it loads, but no pic there. So reported to BrickStore, might expect an update.
Any suggestions, comments, etc. welcomed, here or pm.
The only difference is the price, but they are pretty balanced (biggest difference is 24% compared to the cheapest country/currency).
The PAB-bsx list is in USD but you can pretty much calculate with those prices on the cross rate. If it would be useful, I can make a GBP bsx too, though I don't know how the script calculates.
This program would gain much more usability (for me at least) if there was an option to use it for minifigures. I'm primarily a figure collector, so to have a program that would take the hassle out of trying to find lowest costs across BrickLink would be invaluable.
Just thought I'd throw that out in case someone wants to try and implement this feature.
Another issue is that if too many people are using this, it may well drive up prices as more people Bricklink sets and buy up all the inventory for specific in-demand sets.
I would love to sit down and look at the algorithms it is running, it makes no sense to me why it should take this much time to run. 170 million possible combinations is not actually very many, modern CPUs can do hundreds of billions of floating point operations per second, which of course isn't the same as running combinations that fast, but it shouldn't take 24 hours to run either.
380K bricklink stores, times all the inventory, clearly getting a quick result is all in how the algorithm is designed to run, what it prunes first, etc.
Do you know how much work has gone into making it smarter, rather than just brute forcing it? (which I would agree would take a long time)
My first thoughts would be to sort the bricklink inventory by best price per part, after all that is what we care about most, right?
So you would have a long list of parts in a database, with the lowest price listed first, rather than sort by seller.
Lets just say you want to bricklink UCS Falcon... Picking used, Bricklink gives us a part out value of $1,001.49 including 5,172 items in 267 lots.
So how do we run a search for that?
Some of those parts have over 100 in the set, others have just 1, with lots in between. Clearly some are common as dirt, others are very rare.
If you run the search using lowest price first, you'd probably get 50 different sellers, which of course isn't very useful. Then you need to prune that list, take the seller with the most in stock first, look in their inventory for everything else they sell that is in the wanted list, anything that is within say 10% (user adjustable of course) of the lowest price, add it to the buy and remove from the cheapest seller. Then repeat with the next largest seller in the list.
Will that get it down to 3 or 4 sellers? Maybe, maybe not, but for UCS Falcon, buying it all from 3 sellers is probably not going to happen anyway, so perhaps 5 to 10 sellers is reasonable for that.
This sort would work very well if you did multiple sets at one time, say UCS Falcon, UCS ISD, and UCS Death Star II. With all the overlap between those sets, you'd get more orders worth doing.
Of course, I'm thinking of big orders, even if you had 30 different orders, if you were donig the parts for all three sets, that is 10,000 parts, those orders are big enough to be interesting.
I honestly haven't considered smaller lots, say 10 lots of 100 parts, but I'd think that is small enough to do manually, or the math on it would be so simple, you could brute force it, but I honestly haven't thought about it that much.
I just looked at the Black Pearl which my wife built, same part, but a whole lot less money.
I wonder if you could spray paint the black part to gray and just use that?
Doesn't someone already make custom Lego compatible parts? I see that Brickforge makes minifig accessories, but not bricks. I am wondering how hard it would be to make a replica of the gray boat rigging?
If they could be made for a dollar or two each and sold for $10 each, would there be a market for that if they were close enough?
In fact, this can only be bad for the collectors/builders because then the part prices will go up. I can't see a reseller buying parts and then selling them for the same price or taking a loss.
I BL-ed some dozen of 3300003 recently, without using the $subject, just by using my scripts (to get prices from BL [best 100x lot world, best 1x world, best 1x europe, best 1x local country]), excel and brain.
So some [I believe] useful thoughts for any future manual or automatic algorithm:
- If there are no X (10, 25...) shops/sellers for a part, then it can be considered rare
- If there is no X (100...) pcs lot available at any shop, then part can be considered rare
- Consider getting rares from LEGO "replacement order" (if possible)
- You must NOT look up for the most expensive part, but for the most expensive lots, eg. 100x of something "not-so-expensive" should be prioritized over 1-2 pcs of some "mildly expensive" part
- Start with the "expensive lots" and once the rest lots' prices are lower than a shipping cost, then add those to any existing order (by best price)
I was thinking about making such a program, ended up that it should be an online application for the GUI, then run the queries - to get prices from BrickLink - and the calculation on the server where GUI is running, - likely in the background -, and get back the result online [+downloadable/mail, etc].
Then ended up with the thinking, that the best would be if this was programmed on Bricklink :-]
Consider that your first $20 pledge if you did it.
Regards
Rich
The only condition is that we would need bricklink's approval and a commitment that they would not copy it. In return, we would provide part of the revenue to them to help them make money as well.
However, I recognize that converting the code into an executable would make it available to a larger audience and let it run much faster, but that is beyond my skill set. Matlab has the ability to insert C++ code into its scripts, and converting just a few key functions into compiled code would immediately improve run times. I was hoping some programmers in the community might be able to help in that regard, and donations for their efforts would be appropriate.
That said, trying to monetize this at the expense of buyers in what is already an expensive hobby is not my intent. Current web tools give sellers the advantage; this program tries to balance that by giving buyers information not available anywhere else.
Regarding run times, I think the mathematics involved demand some respect. The heart of the search is an N CHOOSE K problem. When searching a list of common parts, the number of vendors selling those parts (the 'N' in NCHOOSEK) can easily be several hundred. As an example, if you have 300 vendors from which to find the best 4-vendor solution, one must check 330,791,175 combinations. Now for EACH of those millions of combinations, one must sum the quantities for all desired parts over all vendors and compare to desired quantities. Lastly, all combinations that meet quantity thresholds must be product-summed to calculate a cost.
In efforts to limit run times, I have used Matlab's profiler to optimize and have rewritten key parts of the code several ways to see what's fastest. However, the most significant gains are made by reducing the vendor search space 'N'. Therefore, I included tools for intelligently pruning the vendor list to any size (and run time) desired. However, note that ANY method of pruning, sorting, etc. that does not brute-force check all the combinations cannot guarantee with mathematical certainty to find the lowest price. But maybe it's close enough. :)
The only thing I was wondering was if there could be an option to show store results that have most, but not ALL the parts on your list. I have the feeling that one or two of the items on my list are holding me back from getting any results.
I can remove items here and there and re-run the program, but it starts to feel like trial and error which was the original difficulty in manually BrickLinking in the first place. My other option seems to be increasing the number of stores to search to 5 or more, which should take care of a single piece which is holding me back, but once I get to that many stores, the processing time goes through the roof.
Thanks again atxdad. Very cool scripts!
My script also has a feature that will help you identify any rare parts on your list that may be preventing a solution. Set the 'usevendorpruninglogic' flag in merge_price_guide_data.m, and the diagnostics will include a list of how many vendors are selling your rarest parts.
My comments about spending money on development simply have to do with producing an easy to use program that can be run by anyone, that has compute times that are reasonable.
Of course that costs money, so someone has to pay for it. :) I agree with you completely, but 330 Million combinations should not take hours to search, modern CPUs are able to do that kind of math in very little time. Of course running those combinations takes more than doing a floating point math problem, but a modern Intel CPU can exceed 100 Billion FLOPS (floating point operations per second), clearly there is some slow code in there somewhere.
Not your fault, this needs to be rewritten in at least C++, or perhaps something newer such as C#. Assembly Language is of course best, but that takes time and skill, and costs money.
1) MINIFIGURES and SETS - The search can now find those item types in addition to parts. Sticker sheets are now a viable search option too.
2) Faster run times - I found a way to reduce the vendor search space WITHOUT risk of missing the best solution by doing a pre-comparison of all possible vendor pairs before starting the final search. Speed improvements are anywhere from 5% to 90%(!) dependent upon what's in your wanted list.
..plus some other diagnostic and usability features per user feedback. Thanks, all, for your continued interest.
I've nothing new to add. Just wanted to share the love! I predict big things for this script.
Over the past few weeks I've bricklinked up 90% of a MF, for about $600 (with a few small color changes to the superstructure). When Brickstore gives average cost as being over $1k, that's an amazing savings. Similarly, tonight I spent maybe 2 or 3 hours running the parts list for the Eiffel Tower, and I've got a 5 seller solution for $270, excluding only the 250 fence pieces (which I'll get from LEGO Direct). If I remember correctly, that's actually less than the original retail price on the set.
For people new to using this, I've found it most efficient to do some prep on your wanted lists before you begin. My strategy has been to first separate known rare and expensive parts into their own list. I split the remainder of the wanted list into smaller sub-lists of maybe 75 lots. I save vendors in each sub-list's solution, and finally run the total list with the known vendors included, and some extreme pruning of the master vendor list (I aim for around 50-60 vendors).
Occasionally while running through sub-lists, you'll come across parts with very few vendors, or that you need very high quantities of (say, over 200). I've been moving them to the rare parts wanted list, and handling the items on a case by case basis. Sometimes a vendor you're using will have them, sometimes it's easier to find the cheapest available.
The Eiffel Tower was an interesting case of high part counts affecting the price: a 5 vendor solution of the first half of the model came to about $500. With all parts I needed over 100 of (maybe 15 lots or so) moved to their own wanted list, I was able to get the cost on that half of the model down to about $190. What I'm assuming happened is that there was no way to source all of the parts in the first case without using high priced bulk vendors, because of limited slots in the solution (Of course, that's just my guess; there very well may be some other interaction there I'm completely missing).
So you either have to get them from Lego, or find something else that works, or pay that crazy price.
So the question becomes, is that part one that TLG still makes? Will it continue to be available for sale from them?
BTW, original retail for Eiffel Tower was $199, or about the same price per part as Tower Bridge is today.
Given that Tower Bridge is made up of so many cheap parts, I expect we'll see it do about as well as Eiffel Tower and Taj Majal on retirement, and be just as cheap to bricklink if you're so inclined.
I wasn't sure if it was $200 or $300 originally. Still, I won't argue with paying as much as I am, a used complete set is 5-6.
Just though you'd like to know. I haven't tested this with Sets, but I would assume it does the same. Other than that, everything seems to be working great (and faster!)
I swear I've tried to find a method for doing so but haven't been successful.
I'm Bricklinking my first project (Green Grocer) and trying to get the workflow down. I've discovered creating wanted list in Bricklink, searching Bricklink vendors for the wanted parts, and removing items from the wanted list after receiving them. The missing link I have is between the Bricklink wanted list and the Brickstore program (which allows me to create files to feed into this script).
It sounds like I'll have to keep the two lists (Bricklink wanted list and the Brickstore list) in sync manually.
Is there any way to do that? Any way to convert the XML text from Rebrickable into something that Brickstore or these scripts can read?
Thanks
File>Import>BrickLink XML