Automated BrickLink cost minimization program available

Titus

Ah-ha! I can run it in 'demo' mode and that option appears. I always thought that that should be an option and was surprised that it wasn't. Thanks for the hints, dustin!

justmichael

Bosstone100 said:

@justmichael - I think it'd be easier to use Brickstore to keep your inventory instead of messing with the wantlist on bricklink. Once you update your Brickstore file, just copy and paste it to replace the older wanlist. Worked for me quite well when doing Cafe Corner.

Indeed, I think I've finally reached that conclusion. I now consider Brickstore as my "master" wanted list. I do all my editing in there and then export the resulting lists to Bricklink.com. Causes much fewer headaches.

(... and now to find some rare Sand Green bricks, sigh)

Bosstone100

@justmichael - good luck with those sand green 1x8 bricks. When I looked, I couldn't find any. Figured it was easier to buy one complete. They go for $2 I think so bricklinking it may not be the best solution... at least for me it was.

I think a lot of people tried it and available quantities have been depleted.

justmichael

You're crushing my dreams Bosstone.

CupIsHalfEmpty

Wait till September when the Haunted House comes out. It doesn't have 1x8s in sand green, but it does have lots of 1x6 and 1x2s which you could easily sub in if your so inclined.

Bosstone100

Sorry about that. Just warning before you dump lots of money into other parts only to not get the sand green pieces you need.

Titus

Index 53: 53 of 55, 96.364%
Index 54: 54 of 55, 98.182%
Index 55: 55 of 55, 100%
Search for vendor combinations completed in 3052 minutes
No vendor combinations found.

Search completed for this session. Inspect progress of other Matlab sessions
and press any key to continue only when other results are ready. PAUSED
Loading results: multiproc_output_1.mat
Loading results: multiproc_output_2.mat
Loading results: multiproc_output_3.mat
No vendor combinations found.

So sad. Ran for more than two days straight, using 4 of my eight processors and I come up with nuttin'. Oh well :)

Brickarmor

Greetings all. I really want to use this program, however I am helplessly inept when it comes to all things in computer-speak. I did begin reading the thread from the beginning, and to add to my confusion it seems there have been numerous ad hoc changes, so that by the time I understand post X it has been rendered obsolete by post Y... Hope that makes sense.

Soooo I wanted to ask those of you not technically retarded like I am if at this point the procedures are fairly straightforward and user-friendly. I'll be using a pretty new Mac if that info helps. If I could make a wish, I'd wish for a comprehensive and up-to-the-moment post that clarifies current procedures for basic use and provides all necessary links and instructions. If such a post is lurking in these pages somewhere a link or repost would be super.

Thank you!

atxdad

@Brickarmor, The distribution package comes with extensive documentation that gets updated with each released version, plus it comes primed with one example to run out-of-the-box. It requires Matlab or Octave math software to run the scripts (the former is expensive and the latter free) but I leave that for the user to install.

Program development is continuous between the announcements of new versions. There is currently a multi-user behind the scenes effort to test compiled versions of the code that run hundreds of times faster than any previous version. Those will be included in the next release after an appropriate vetting. To receive a copy of the latest scripts, PM me with an email address and I'll send one along.

canon03

Thanks @atxdad for the code. I was able to run it this week and take advantage of the multi-core capabilities. This program really does save you money, and doesn't take that long to figure out.

atxdad

For those following development, I have now assembled another distribution package for the public that is the most significant improvement yet. For common computing platforms, the program should now run HUNDREDS of times faster than previous versions, mitigating the most common barrier to its use. I have also added a new vendor filtering option to help automatically identify and exclude vendors that are consistently priced higher than what you, the buyer, considers a fair price.

The amazing speed increase is due to the efforts of many BrickSet users who helped compile and test this new version, but special thanks go to user @CrazyP who voluntarily translated key Matlab functions into C++ source code. The tool is now truly a community effort.

Titus

Thanks atxdad!

Mad_Dog

Great atxdad, it is a great tools to help us.

drdavewatford

Can I just say what an unexpected, but amazing, use of the Brickset Forum this community development is ? Anyone who has struggled to figure out which combination of Bricklink vendors will provide the most cost-effective solution will immediately see the value of this project. I just look forward to the day when it will be available as a simple, standalone application, at which point I'll use it like a shot !

Bravo and thanks to everyone involved.

USMCYodz

Just used the scripts for the first time. Absolutely fantastic and a lot easier then I thought it would be. Hardest part was downloading all the software to get it all going. Bravo @atxdad you and everyone else who has helped have done a outstanding job.

My one question, is there any spot in the script to include the "Min Buy Amount" specified by some vendors? Seems a few of the choices I am given are always lower but don't meet vendor requirements. Could this also be done for shipping as well? I am sure bricklink has that data in the backend somewhere, I just don't know if you could grab it like the script does for part prices.

atxdad

@USMCYodz I've considered those requests before but don't know how to support them. Shipping costs have too many dependencies, and I don't think they are saved anywhere within BL as a metric. I have seen BL report a vendor's minimum purchase amount when viewing one's wanted lists, but I don't know how to parse that information generally. If someone knew how to easily request that info I could probably add a check at the end. Recognizing these limitations, I show several solutions so you can estimate shipping for all and calculate the best deal.

FourMadMen

This sounds like the perfect thing for me an a large project I have. How does someone get the distribution?

andystar

^ I believe atxdad will set you up if you send him a message

jarom_moroni

Wow! This is amazing how so many people have come together to benefit the community! Major props to @atxdad for the great work! As a former seller on BrickLink I would totally support this kind of thing. It is this kind of innovation that moves markets forward, increases competition, and in the end causes sellers to respond by creating their own innovations to bring prices down and increase the value they give to their customers. This is a very healthy function of the free market and this kind of innovation drives it! Great job @atxdad and all others who have helped! I LOVE IT!

KyPhinfan

First, thanks to atxdad and everyone else who has worked on this tool to make it better.

Second, I may be a little slow, but where can I get a copy of the tool? I realize I'll need to download Octave but can someone point me in the right direction?

Joseph

andystar said:

^ I believe atxdad will set you up if you send him a message

Bobflip

Got this going yesterday, seems to work pretty well! I'm in the UK, so changed the priceindstart line, and that works great. Was pretty fast in pricing up the test 7110 wanted list. Very good job on this :-)

The main thoughts I have atm, are that it can be expensive shipping from the US/Australia to the UK though, so am wondering about ways to filter out all non EU stores. In the UK, there are three postage bands: Within UK, Europe, and Rest of World.

At the moment I'm thinking of a flag (or perhaps two) that we can set within one of the scripts to select either our own country or choose one of the other two options. I suspect this would be similar for other countries in the EU, but don't know how international postage bands are set in the rest of the World.

Any thoughts on this?

mshiels

For sure the fastest way to find decent matches/least number of stores is starring from/downloading the matching stores for your want list - I bet within a certain number of stores you can find a pretty decent result.

You can possibly temper how far it has to hunt by things like average prices for parts etc, as long as it is in the right ranges, no need to hunt further.

atxdad

@Bobflip The filtering of vendors by nation is my most frequent feature request, but everyone has their own favorite flavor, i.e UK only, EU only, Latin America only, etc. The ideal universal solution would have the user first provide a list of acceptable vendor countries. The program would then prune the vendor search space to just those countries and conduct the search.

The US-only filtering option is implemented not by parsing vendor info, but rather by checking if the price is reported in US currency. To query and retain vendor location info would require some extra parsing and bookkeeping be added to the code, but it should be possible. In the meantime, one way to limit your vendors by location would be to prune all vendors (with the vendor yield filter, for example) while loading a personal white list of vendors from your location. See the vendor pruning logic in merge_price_guide_data.m.

tsrplatelayer

Having trouble getting this to run - advice would be appreciated.
Using Win 7
Installed Jave Runtime
Installed file 7z utility - watching out to only install the decompressor and not the rest of the rubbish it wanted to install.
Downloaded Octave
Downloaded Octave _pkgs
Installed octave in c:\octave folder
Installed Pkgs to c:\octave folder
Ran Octave and rebuilt as per instructions.
Octave now loads, runs through the startup script, with many warnings, then gives prompt,
Octave responds correctly to simple commmands i.e. pwd()
Now how do I get 'run_all.m" to execute?
Cost Minimisation code is currently loaded in c:\lego folder - where should it be?
I tried:

cd c:\lego
source("run_all.m")

program started then gave a message:-
panic: segmentation violation - - stopping myself

Advice appreciated

Allan

tsrplatelayer

Well - I removed Octave 3.6.1 and replaced it with 3.6.2. Now the cost reduction program starts without crashing Octave. But there are lost of error messages

I didnt make any changes to any files just 'run from box'
The following output appears
Reading wanted list Wanted List for 7110 Landspeeder.bsx
Warning: load: can not read non-ASCII portions of UTF characters; replacing unreadable characters with '?' (abt 50 of these - didnt count them)
Requesting data for 1 of 23: Any Coplor Luke Skywalker (Tatooine)
warning: C: \lego\readpricequide.m: possible Matlab-style short-circuit operator at line 32, column 20
warning: C: \lego\readpricequide.m: possible Matlab-style short-circuit operator at line 35, column 24
warning: C: \lego\readpricequide.m: possible Matlab-style short-circuit operator at line 76, column 32
ERROR during Price Guide parsing, likely due to currency mismatch
Non-US users, see README notes for instructions on making modifications.
error: Stopping
error: Called from:
error: c:\lego\findnextstring.m at line 12, column 5
error: c:\lego\parsepriceguidestring.m at line 36, column 14
error: c:\lego\readpriceguide.m at line 65, column 14
error: c:\lego\parse_wanted_list_and_price_guide.m at line 90, column 58
error: c:\lego\RUN_ALL.m at line 13, column 1

and prompt reappears.

I will have a go at the currency changes and report back.

tsrplatelayer

OK - i edited the currency setting and non-US setting and - TADA - its working

Hope my experiences ar eof benefit to other users

Allan

tsrplatelayer

restricting result to area/country.

tsrplatelayer

Oops - pressed enetr - silly me - now where was I..
Oh yes - I had an idea.
If this script was configurable so that the user could enter ther Bricklink id/pwd into this script then the script could then log in to BL as if it was the user. Provided the user has configured there BL account properly then when you go to My Wanted List/By Store it only lists the stores in the country(region) the user has already preselected.
This page could then be parsed out to provide a 'white list' of stores to use for minimisation.

Allan

Bobflip

@atxdad Thanks for the info and tips, I'll have a play with those in the meantime.
One question about the US-only filtering being done by checking currency, aren't there some stores that sell in US dollars but aren't in the US? Or am I mistaken about that?

atxdad

@Bobflip Yes, I know of some non-US vendors selling in US$, but I haven't ever encountered any in my normal usage of the tool. If it's a problem for you, just enable vendor pruning options and blacklist the vendor(s).

djomp

I've got an issue trying to change to GBP; I edit the parsepriceguidestring.m file as detailed in README but get the following:

??? Error using ==> findnextstring at 12
Stopping.

Error in ==> parsepriceguidestring at 37
priceindend=findnextstring(temp,'',priceindstart+1)-1;

jack_bricks

good evening. may I ask those of you who use the programm a lot how do you deal with the rar parts? I mean to get a solution i usually search up to 5 vendors BUT at this stage and above it'll take forever. Because buying 10 rar parts only sky rockts the shipping costs.

graphite

In some cases the rare parts just prevent it from finding a good solution. The essence of rare parts is it makes it hard to get them. The best way to resolve it is to remove the rare parts from the search and let it find a solution for the rest of your list and order the rare ones manually.

atxdad

@jack_bricks If the search for a solution is taking too long, you might consider limiting your vendor search space, but in a way that preserves the vendors of rare parts. When you enable vendor pruning logic, one of the diagnostic tables created sorts parts by rarity. You can set a rarity threshold that will guarantee the vendors of those rare parts are included in your vendor search space. Then use the remainder of the pruning logic to reduce the rest of the vendor search space down to a more reasonable size.

And if you haven't done so already, consider allowing USED rare parts instead of just NEW; that may open up a few more vendors for consideration.

graphite

@atxdad is there a way to get more information about a store when results are printed? The search I'm working on right now ends up with a vendor that appears to have spaces/blanks as the store name and there appear to be hundreds of stores with no name. Would it be possible for the results to include the username or link to the store?

atxdad

@graphite The code currently saves only store name and whether it is a US vendor or not. Part of the planned changes for a next update include filtering on vendor nation, so that information may be available in the future too at search end. I think there are several blank names in the vendor data file that arose from the initial vendor database construction, but those shouldn't show up as a search result unless blank names are appearing in the Price Guide data too. PM me and we'll debug further.

legolijntje

I tried the program; and it works like a charm (although very slow, but I have a file with 207 lots :P).

I maybe want to try to make this, using your code as reference, in Visual Basic, so one can get a "real" aplication, but I'm very, very new to visual basic. It's more like an idea and maybe a future project :)

Anyway, thanks for the code :D

rock99rock

I've been looking for something like this! One thing though, I don't see where to download it from? Are you taking donations yet atxdad?

Joseph

Joseph said:

andystar said:

^ I believe atxdad will set you up if you send him a message

:-)

graphite

@rock99rock I don't think he has a download link. Just send @atxdad a PM and I think he'll email it.

atxdad

@rock99rock Yep, just send me a PM and I'll forward the latest distribution package to any email address you provide. Thanks, but donations aren't necessary. I do this freely as a hobby.

For any still following along in the development, my notes show that I've had 170 requests so far. I doubt that many are using it, but I'd be curious if any use it regularly. At least I do!

Tevans333

I use it regularly. It's been incredible from the start, but especially since the compiled code was added. I can't imagine using brinklink ever again without it!

emilewski

I do not order from BL often, but when I do I always use it. My CC project took huge advantage of it (unfortunately that was before the compiled code update so it took a LONG time). It is a great tool! I anticipate in a few months having some large BL projects and will definitely be using the program for that. Thank you for your wonderful tool!

Bosstone100

@atxdad - I use the utility all the time. I create .bsx files for missing parts on sets from my childhood and run the utility to do the searching for me. I probably wouldn't use bricklink otherwise.

Mad_Dog

I do use it as a guide and help fir BLing cost estimation and purchasing, great job!.

Rgrds

Combee

I'm in the same boat as emilewski, I don't order from BrickLink that often but when I do I always use the program. It is invaluable and I really appreciate all the work that you've put into it!

rocao

@atxdad: I still use it regularly. The last time was this past christmas to source parts to build ornaments for family members. I owe you a PM still... I wanted to take the time to respond thoroughly and thoughtfully.

mdelleman

This is what I am getting when run in MatLab

>> RUN_ALL
Reading wanted list Inventory for 4000001-1.bsx
Requesting data for 1 of 182: Dark Bluish Gray Plate, Modified 1 x 2 with Handle on Side - Free Ends
WARNING: No sellers found for Dark Bluish Gray Plate, Modified 1 x 2 with Handle on Side - Free Ends
Program stopped. Suggest revising wanted list.
Error using load
Unable to read file Price guide data.mat: No such file or directory.

Error in merge_price_guide_data (line 89)
load(priceguidedatafile)

Error in RUN_ALL (line 14)
merge_price_guide_data



There are certainly sellers for that item, not sure what is happening. I have changed my currency to CA but still nothing. Any ideas?

mdelleman

^ Looks like I had to clear the US Flag as noted in the README (for shame).

Up and running now, what an awesome tool!!! Thanks @atxdad