Automated BrickLink cost minimization program available

atxdad

I am a frequent user of BrickLink, and often after creating a Wanted list, I would find no one vendor with everything I wanted in stock. The resulting manual search for the least expensive mix of vendors to purchase from was time-consuming. Therefore, I wrote an automated tool to do this search for me which I'd be happy to share.

For a given Wanted list, my program downloads BrickLink pricing info for all desired pieces and does a search over all possible vendors to find the combination that fills the order at the least cost. It can also filter on New vs Used parts, specific or undefined color, US vs international sellers, and split partial quantities over multiple vendors if desired totals can be met.

My solution is a package of Matlab scripts which I can share with anyone who asks; just send me a message. The tool does require Matlab, but I'm probably not the only Matlab-using engineer in this demographic, and I'd welcome any effort to make this into a stand-alone executable. The example for parsing a Brick Link Price Guide page might be helpful to programmers on its own, and I'd also be interested in hearing about any other solutions that solve the same problem.

StuBoy

Sounds extremely useful. I have no idea how to program so will be hoping for a stand-alone executable. Good job and good luck!

atxdad

One of the admins pointed out they have a 'no robots' policy that might apply here. My script doesn't hit the site more often that I was doing manually, but it is automated. I asked for their permission to use it; we'll see what they say.

Si_UKNZ

sounds like a brilliant idea. why they havent implemented this themselves I really dont know.

parkerwilson

sounds like a brilliant idea. why they havent implemented this themselves I really dont know.

I think if BL implemented this themselves, it would upset alot of sellers. Lots of sellers that have a large assortment of items, tend to price a few cents higher because people will purchase anyways since it can be a one stop shop kinda deal.

stephenvw

Would love to be able to use a program like this! Please keep us updated on what they say.

dougts

I've sent you a PM atxdad. This is a fantastic and highly useful idea!

DaddyWhale

sounds great! How does your program deal with shipping costs, in particular fixed
Shipping costs (like "min shopping cost is $

atxdad

Given the variability in shipping options and costs, I only minimize on the nominal cost of parts, but offer solutions for 1, 2, 3, ... vendors. The user can then decide if the additional shipping costs for adding an extra vendor is worth the savings in parts costs. Just minimizing the number of vendors isn't always the best solution, but often is.

DaddyDeuce

I'd love to see them. I don't have access to MatLab, but I was thinking about writing the same sort of then in C or Perl. Seeing your source code could give me the kick-in-the-pants I need.

bluemoose

I used to, in a previous post, have access to Matlab, Simulink & pretty much all the toolboxes, including the Compiler, but not any more :-(

Some m-files will run with only minor alteration in Scilab, the free Matlab-a-like alternative. Not sure if it supports web access though. Might be worth a look?

Coolsplash

Excellent work @atxdad! Do keep us updated on this.

andystar

I believe another alternative to MATLAB is Octave. Although I haven't used it in awhile, it was fairly compatible with MATLAB m-files. Not too sure about the web access too.

I would very much like to use the tool @atxdad wrote, but need it to run on free-ware.

atxdad

Update: I haven't received any feedback from BrickLink admins and won't try to interpret what that means, but I would point out that the most important component, the cost analysis, is distinct from any BrickLink automation and can be used independently offline. I only added the latter after tiring of manual data entry.

I'll also share a couple anecdotes from development that illustrate the utility of the cost analysis. In preparing to make a $100+ order, the algorithm found me a solution that was 20% cheaper than the best solution I could find manually. I have also found examples where it is 40% cheaper to buy a set by pieces instead of as a whole. I think this is a tool buyers need because the calculations exceed human ability and the savings are potentially significant.

dougts

the big time BL power sellers will hate it, because they make their money by being "one stop shopping" and not by having the best (or even at times competitive) pricing item to item. A tool like this could take away a lot of their highest-margin items.

but for the buyer - it's a huge win.

effalcon

sounds like a brilliant tool.

DadsAFOL

the big time BL power sellers will hate it, because they make their money by being "one stop shopping" and not by having the best (or even at times competitive) pricing item to item. A tool like this could take away a lot of their highest-margin items.

but for the buyer - it's a huge win.

As a major US BrickLink seller, I actually support this tool. We put a lot of effort into maintaining our pricing around the price guide average. There has been debate for years about the fact that the price guide link is suppressed from within a store. Supposedly this keeps buyers from simply finding the lowest price for an item and a seller "losing" a sale to another store with a lower price. Which seems a bit silly to me, since I just open a second browser window to the price guide. Reputation and customer service are equally important decision points for BrickLink shoppers. When I'm buying, I will pay a little more for something if I know I can get it from a store that will ship it immediately and professionally (i.e. sorted and bagged).

I haven't tried the Matlab program yet, but I'm definitely interested.

jocappy

Can't wait till something like this is available for a non programmer like myself. Nice work.

Si_UKNZ

I must admit that every time I see this thread title it make me think of a 'program' in the sense of 'bricklink rehabilitation program', and I'm thinking "yeah, I'd like to hear more about that!.

Sounds like a great tool, so +1 to what @jocappy said ^

PS nice to see a big bricklinker on here @DadsAFOL.

andystar

I've successfully run @atxdad's MATLAB program with no modifications on Octave, which is a free MATLAB emulator that runs on Windows.

If you can install software and click through a few buttons then you should be able to do the same. For those interested do a Google search for it, also get the Octave GUI (again google search). You'll need to download a zip utility www.7-zip.org to unpack Octave, its free too. Once this is installed you can launch the GUI executable and tell it where the Octave binary is located, this way you have a nice GUI to run from instead of just a command line (which can frighten those not familiar with command line stuff)

Please note running the scripts is not a simple executable file, there is some typing involved (>> run RUN_ALL.m ) and if you want to change things from the default you'll have to do some basic text editing on @atxdad's scripts.

Out of the box the current code creates plots, but after I disabled them my runtimes of ~10-20min for a 23 item part list that searched across all 1,2, and 3 vendor combinations for the lowest price.

DaddyWhale

Thank you for this. Very very helpful!

atxdad

Update: Based on helpful feedback from the initial users, I've made several changes to the cost analysis component and prepared a second package for distribution. Some specific changes:

-Optimizations to decrease run time by 25%
-Better compatibility with Matlab emulators
-Notes for non-US buyers
-Additional analysis tools that try to reduce run times by intelligently limiting the search space

Experience has shown that an exhaustive search for a least-cost solution can exceed practical run times when the number of vendors to consider is too large. (An NchooseK problem explodes.) In those cases, new analysis tools can be enabled that try to guide the user in pruning vendors from that search that would likely not be part of a final solution anyway. Doing so can potentially find a solution that would otherwise take too long to compute but without the mathematical certainty that it is the best solution.

Thanks for the comments so far, especially from those who have given it a try.

caperberry

Spookily i posted on a forum yesterday saying how nice it would be to have something like this! Dreams really do come true.
Except I'm not a programmer so will just have to wait :O)
Amazing you're getting 40% discount against some sets!!

LegoFanTexas

the big time BL power sellers will hate it

Will they?

One issue is shipping costs... Getting everything for slightly higher prices, but from far fewer sources, can actually be less expensive than ordering 3 parts here, 6 parts there, etc.

True, if the prices aren't even reasonable, that is a factor, but there is also the issue of placing 10 orders and keeping track of those, vs. placing 50 orders and trying to keep them all straight.

$300 to put together a set with just a few orders vs. $200 to try and put it together from dozens and dozens of orders, seems like a poor use of time.

StevenAshby

^$100 dollers whilst it sounds like small change to you is a lot of money to a lot of people, to them the price saving is easily worth the time to get more for there money.

Legogeek

First, I'd like to say thanks to @atxdad for working on such a useful tool.
Unfortunately I haven't a clue to what you guys are talking about with the tech speak, so I really hope this eventually can be made useable by the most software illiterate users like me! ;oP
There's a few smaller/medium builds I'd like to try but haven't bothered with because of the piecing it together aspect - so I think this would make things much easier.

I'm okay with paying a little more for pieces if I can get them from as few sellers as possible, but I realize other collectors have just the opposite view. There is no right way or wrong way - we're all free to choose our own way. The important thing is getting what we want. :oD

AvengerDr

I have tried to run it within Octave (through the windows octave GUI) but even after configuring the proxy it says:
“error: urlread: curl: HTTP response code said error”
(you don't say! :D)
any clues?

cheshirecat

Sounds like a brilliant and very useful tool.

Unfortunately MATLAB is horrible when it comes to creating executables or DLL's. Last time I had to do it you still needed to install a 200MB 'mini-MATLAB' and then depending on other things on your system you had about a 3in4 chance of it working properly. Eventually I gave up and just coded all the maths from scratch in C and dumped MATLAB once and for all. But I suspect that wouldn't be a trivial thing with this tool.

However, if it can be run through free versions of MATLAB-A-LIKES then that sounds like the best route for those without MATLAB. Maybe with some simple user guide to walk through the steps?

markg11cdn

I haven't 'upgraded' to the new version yet, but I just finished using the scripts to find where to get the parts for a new caboose. If you have a list of relatively common parts that are found at multiple vendors, this program will find the one with the lowest price for the total parts. Saves you from having to go to each vendor that matches your wanted listed and manually checking the prices.

For my 30 lot/135 piece list, the 2 vendor price was $29.70. The 3 vendor solution is $15.23. Adding another vendor will probably add another $2-3 in shipping, so I'll still save around $10 but using 3 instead of 2 vendors.

Here's also what I've done to let me use the program in Canada against US/Canadian vendors:
1 - created a short script, getstores.m that went through all ~380k vendors in Bricklink and populated the vendor_locations.mat file with those that were Canadian or US
2 - modified the parsepriceguidestring.m
a) remove the check for '~' in the price
b) change the text 'US' to price 'CA' for the 'priceindstart' findnextstring call

Thanks for the great work atxdad!

cheers - mark

burnside

@atxdad Great job, I'm definitely going to try this out.

I've never programmed anything in matlab. Just wondering why you chose that for your language instead of something else. If this were a web app in PHP/Javascript, I could see more people using it.

atxdad

@burnside A fair question. I've been using Matlab daily for 20 years, and it's really fast for prototyping. I also don't know any other languages, so that kind of limits my options! I don't mind doing the algorithm development and then handing code off to others for implementation, hence the invitation for someone else to make an executable.

burnside

@atxdad I know PHP/Javascript in and out. I could take a look at the code and then see if I can port it to a web language.

cheshirecat

^ If the processing is quite intense a web app probably isn't the best approach. A desktop app written in c would probably be better.

AvengerDr

I tested it outside my university's network and it seems to work (although it stills complain about the UTF encoding). @markg11cdn could you share your "getstores.m" script so that I could try to edit it to include the UK and some other EU countries?
Thanks!

bkpr

I'm in the same boat as legogeek; I don't mind paying more to make the process of buying multiple parts easier. This would be incredibly useful tool for me.

Anyone care to port the (hopefully coming) desktop software to Mac? Pretty please?

graphite

@atxdad is there a way for it to take into account the minimum buy ammount from the vendors?

dougts

been using this off and on the last couple days. it definitely is a valuable tool, but takes some tweaking when working with large lists and/or more rare parts.

I finally ended up pulling all my rare parts off my list onto a separate one, buying those items first, along with whatever else I needed to buy to meet vendor minimums, then removing those items as well from the main list, and running the tool with what was left. worked great - ended up finding a 3-vendor match for a 46 lots, 384 piece wanted list. there were 2 cheaper 4-vendor matches as well, but the difference wasn't enough to cover the extra shipping charges, so I went with the 3.

atxdad

@graphite, the tool does not account for a minimum buy requirement from vendors because that information isn't on the Price Guide, and I don't know how else to query that data. However, the tool reports the price/quantity for a given part for ALL the 'final solution' vendors, even if desired quantities can be met by just one of them. With that information, some human judgement could shift common pieces from one vendor to another to meet everyone's minimums, should that ever be a problem.

atxdad

I have received several questions from those wishing to use the tool to prepare a 'big' order. While I think the tool's default settings will work well for orders up to ~50 lots, the following strategies may help find a solution for larger orders. From the algorithm point of view, 'big' means many unique lots; total piece count is irrelevant.

When considering a large order, the first step is to use BrickStore. Use the Edit/Price/Set to PriceGuide/Current Inventory Minimum feature to see the theoretical minimum cost for your wanted list. If this exceeds your budget or cost of a set bought whole, stop here. The tool won't help.

Two common problems to mitigate:
I.) No combination of vendors satisfies the wanted list requirements
II.) Computation time is too long


I. NO SOLUTION FOUND:
If the tool could not find a solution, the likely causes and remedies are:

1) The list of vendors to consider is too small - Make the wanted list less restrictive. Within BrickStore, change some parts' quality from NEW to USED (The algorithm treats 'used' to mean look for both.) or change some colors to N/A (don't care).

2) The maximum number of vendors to combine is too small - By default, the tool searches for combinations of 1, 2, or 3 vendors. Increase this maximum by setting maxvendorstouse=4 (or more) in find_minimum_cost.m.

3) A part on the wanted list is rare or desired in a quantity too large - If a part is sold by only a few vendors, one of them MUST have a slot in the final solution, reducing the flexibility of the tool to fill the remainder of the order. The same is true if few vendors have sufficient inventory of a part. Make a subset wanted list of only the rarest part(s); use the tool to find a solution for the subset and then again for the remainder of the wanted list. (see @dougts above)


II. LONG COMPUTATION TIME
The program's run time is dependent upon the number of vendors being considered. As a wanted list grows, the number of vendors selling any part on that wanted list also grows. Furthermore, every additional vendor to include in a solution (maxvendorstouse) increases run time by at least an order of magnitude. The following suggestions keep the vendor list constrained even when the number of wanted lots is high, and these can be tried without having to pull pricing data each time (skip parse_wanted_list_and_price_guide.m).

1) Apply the minimum quantity filter to prune vendors selling only a few of an item that is needed in large quantities. See the 'minimumqty' vector in merge_price_guide_data.m.

2) Apply logic intended to remove vendors with few desired lots available. Enable by setting 'usevendorpruninglogic' flag in merge_price_guide_data.m. Accept the recommended defaults for pruning thresholds to reduce vendor list by ~50%. Safeties prevent the pruning of vendors selling rare parts or a high percentage of total desired parts.

If the default pruning still results in excessively large vendor lists (>300) and run times, aggressively prune the vendor list to <100 by manually entering pruning thresholds, overriding the defaults. Plots will guide threshold definition. Run find_minimum_cost.m to see if ANY solution can be found in a reasonable time, even if not necessarily the least cost. Iteratively repeat with a larger vendor list (less pruning), until 1) a solution is found at a satisfactory price or 2) run times lengthen to intolerable levels.


In conclusion, I think casual use of this tool will not require the level of human oversight described above, so one shouldn't be intimidated by these instructions. However, there is clearly a population of motivated buyers who want to push its limits. I hope this helps!

Steverd

Wow, I have a BL store and love this idea..

It would be so awesome if someone programmed something like this with a nice GUI and as a stand alone program. I would pay $$$ for it!,

Cam_n_Stu

Fantastic idea!

I was trying to get work this manually with two wanted lists last night and even with the small 9 lot list it was a PITA. I just gave up on the 180 lot list. This and the variable shipping costs and minimum order limits are why I don't buy from BL.

Now if Rebrickable.com could just embed this technology and BL started paying referrals to off set their costs the problem for most people would be solved.

atxdad

Update: I've prepared another distribution package with changes aimed at providing the user with lots of useful internal data that was previously hidden. Some examples:

- The cost analysis can now show detail for the 'Top 10' least expensive solutions instead of just the single best solution
- Total cost/lots per vendor is shown to quickly know if minimum purchase/maximum lot terms are met
- Rare parts that may be preventing a solution are identified

Often there are multiple solutions within a few cents of each other, and I would consider all viable options. Now the user will have enough information to screen roughly equivalent solutions for all of the following before making a purchase decision:

- Minimum purchase/maximum lot requirements
- Shipping costs
- Feedback ratings
- Country of origin

I'm next considering ways to ease turning a hypothetical solution into a real purchase, maybe by generating store-specific wanted lists, etc. Ideas are welcome, and thanks to those who are giving it a try.

graphite

I'm not sure how Matlab/Octave handles threading but I've noticed that the processing is only using 13% of my CPU which equates to 1 core of a quad dual core processor. Something to investigate might be a way to allow it to fully utilize the CPU.

atxdad

That's a limitation that has frustrated Matlab users for a long time; the 'parallel computing toolbox' is available for extra cost. However, one can run multiple Matlab sessions simultaneously. I'll do so using slight variations of a parts list on different cores, or dedicate different cores to a 3-, 4-, or 5-vendor search. I've hit close to 100% utilization before that way.

AvengerDr

Do you still have to change the currency symbol for extra-US countries in this latest release?

atxdad

I didn't make any changes to the parser logic last release, so international users will still want to check if currency prefixes need to be changed. I've put all I know about currency conversions in the README, but that is still functionality I can't check myself.

atxdad

Update: In the discussions of this tool that have spilled into other posts, I've learned how people are really trying to use it, so I prepared another distribution with new features proposed entirely by others. The two most significant additions are:

* Automated creation of vendor-specific Wanted Lists that can be uploaded into BrickLink
* Ability to blacklist/whitelist specific vendors from the search

The latter may seem trivial but helpful in exluding vendors that waste run time with huge inventories of available lots at unreasonably elevated prices. However, the ability to forcibly INCLUDE vendors in the search space is vital to finding solutions for massive parts lists in a reasonable time.

Specifically, a large wanted list can be broken into subsets which search relatively quickly. All those intermediate solution vendors are then forced (whitelisted) into a reduced vendor search space for the original wanted list. At least one solution is guaranteed to be found for the original list, and possibly one at less cost than the sum of the subsets, within a maximum run time you can control by pruning the vendor search space.

Thanks to everyone who continues to send suggestions!

doriansdad

I got heavily into bricklinking a few years back. Alot of the best deals are had from stores that are not open to the general public. Make some contacts on the boards there and this will allow you to unlock some really useful stores and excellent prices and expansive inventories.

Cam_n_Stu

"Alot of the best deals are had from stores that are not open to the general public."

What is the logic in sellers operating like that?

111ins

Would anyone that is much more proficient at using software like this be willing to run the program for someone? If I were to give you a set or two to put thru the system and then you email me the store list?? I have figured out most of Bricklink but there is no way I could figure this out....
Let me know if anyone would be willing.
Thanks, Brian

atxdad

@111ins Sure, I'll try it for you since I'm always looking for new data to test and debug with. That way we both benefit.