Brickset ForumHome›
Shopping at LEGO or Amazon?
Please use our links: LEGO.com • Amazon
As an Amazon Associate we earn from qualifying purchases.
Please use our links: LEGO.com • Amazon
As an Amazon Associate we earn from qualifying purchases.
Automated BrickLink cost minimization program available
Shopping at LEGO.com or Amazon?
Please use our links: LEGO.com • Amazon
Recent discussions • Categories • Privacy Policy • Brickset.com
Howdy, Stranger!
Categories
- 26.5K All Categories
- 16.1K General
- 3 Announcements
- 3.1K Collecting
- 3.7K Everything else LEGO
- 1.7K Building and Techniques
- 2.3K Brickset
- 14 Ambassador
- 1K Introduce yourself
- 380 The Database
- 436 Brickset.com
- 314 Forum Operation
- 15 FAQs
- 13.9K Shopping
- 1.7K Shopping USA/Canada
- 336 Shopping Elsewhere
- 11K Marketplace
- 1.2K Other
- 570 Community and Events
- 262 Photography/Video
Brickset.com is a participant in the Amazon Services LLC Associates Program, the Amazon.com.ca, Inc. Associates Program and the Amazon EU Associates Programme, which are affiliate advertising programs designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.
As an Amazon Associate we earn from qualifying purchases.
Comments
Please download octave here
http://sourceforge.net/projects/octave/files/Octave Windows binaries/Octave 3.6.1 for Windows MinGW installer/
Choose Octave3.6.1_gcc4.6.2_20120303.7z
Decompress with 7zip or winrar at root
Ex: C:/Octave
Then download octave gui here
http://guioctave.com/?page_id=2
Choose GUIOctave v1.6.0 [7.86Mb]
Launch octavegui
Give it where octave.exe is
For instance c:\Octave\Bin\Octave.exe
You have ready in task bar of GuiOctave
Decompress latest version of scripts in c:\octave\apps for instance
Choose in adress bar the folder
c:\octave\apps
Type RUN_ALL
Where does the output go exactly?
And the final output should go to a *.txt file as well as it should print in the Octave window.
As noted earlier, Matlab does not inherently exploit multiple CPU cores which are so common these days, but in my tests, simultaneous Matlab sessions do seem to run on separate cores. Therefore, I automated the division of a large search into smaller jobs that each run in its own Matlab session, and when complete, all results are concatenated into one final report. I can see total CPU utilization of 100% with this modification, and testors confirmed it works with Octave too.
Lastly, for completeness, the search can now support the inclusion of LEGO Pick-A-Brick as a candidate vendor. It requires some manual data entry to use, but it's there now for those seeking a true least-cost solution.
Thanks for the continued suggestions and especially the code samples for me to copy!
The script prompts me the following:
Multi_core CPU processing enabled. Instructions:
1) Start 2 additional Matlab session(s) if not already running
2) Ensure all working directories are hte same as this sessions
3) Run the following 2 command(s) in the other 2 session(s)
In session 1 run: find_minimum_cost_remote(1)
In session 2 run: find_minimum_cost_remote(2)
I'm good up until step 3. Whenever I try to run the command it tells me the script doesn't exist. Can someone please let me know how the command should look? I'm typing "run find_minimum_cost_remote(1)"
1) Verify that the files 'find_minimum_cost.m' and 'find_minimum_cost_remote.m'
both exist in the same directory
2) In the two additional Octave sessions, do this:
a) Use the 'cd' command to point the workspace to the directory verified above
b) Use the 'dir' or 'ls' command to verify the scripts are in your current working directory
c) Run these commands:
which find_minimum_cost.m
which find_minimum_cost_remote.m
They should both list the current working directory
If any of those verifications fail, PM me the screen messages and we'll try some more.
Maybe an Octave user can confirm that you have launched the functions (not scripts) with the correct syntax.
@graphite, you are right, I was typing run and I didn't need to be. Now I'm happily running @atxdad's scripts on multi cores:)
After running it on 4 cores it completed a 4 vendor, 33 lot search in about 2.5 hours, very happy with this! But at the end it did not save to the results to the "Minimum cost search results.txt" file. The script prompted me to wait untill all sessions had completed. I did this and made sure each had a command prompt. I then hit a key to continue and have the results saved, only problem is the results file only has this:
Checking 1.06133e+009 combinations of 4 vendors for those that have desired quantities...
Search completed for this session. Inspect progress of other Matlab sessions
and press any key to continue only when other results are ready. PAUSED
Loading results: multiporc_output_1.mat
Loading results: multiporc_output_2.mat
Loading results: multiporc_output_3.mat
I made sure each session was working in the same directory. Is there a way to generate the results without running the 2.5 hour search again? What may have gone wrong?
This is probably a good time to say that any run-time questions like this can be sent to me directly. Remote debugging is sometimes an iterative process, and I'm guessing few would be interested in my whole conversation.
I can supply the parts list as BrickLink XML, .bsx, .ldr or .txt
Any help gratefully received!
Running list of 115 batches, 3091 pieces, in 2 different files of 37 batches / 837 pieces and 78 / 2254 goes more or less OK with default options, of course 4 shops is looooong calculation, and not always successful, other searches are much better, so keep low batches and merge after, that is the best option.
Atxdad, I am happy to keep checking any new version you may need, keep the good work.
Rgrds
J.
The parts in my order were not expensive but I had 140 different lots, which meant I had to buy from at least 3 sellers. But it turned out buying from 4 vendors was cheaper, even after adding (estimated) postage manually (which of course cannot be calculated automatically).
As the program cannot limit by country/territory, I went through the many output options provided and rejected non-European stores; there were still several options for me to choose from.
The program helped me realise I would have to compromise in some way, which I did by having some Used parts instead. Not an easy decision, but great to be able to make it fully informed.
Now using BrickLink the usual way is just depressing! - so much effort just to know that you are getting a decent price, and it feels like BL is heavily weighted in favour of the seller. A program like this is the way forward and I hope @atxdad has continued success developing this and getting it out there. If BL wants to stay successful they'd be wise to integrate this program into the site, before someone else just does it elsewhere...
Thanks
M.
Read here how to do that: http://www.btabdevelopment.com/ts/default.aspx?PageId=13"
I've changed the numproc value to 8 in find_minimum_cost.m but I don't know how to open new sessions of Octave.
And now, I'm receiving this error:
>> RUN_ALL
Reading wanted list 10179-bb.bsx
Requesting data for 1 of 41: Light Bluish Gray Technic, Pin Connector Round (Pin Joiner Round)
warning: E:\Dev\Octave\apps\readpriceguide.m: possible Matlab-style short-circuit operator at line 63, column 32
error: findnextstring: mx_el_ge: nonconformant arguments (op1 is 1x41, op2 is 0x0)
error: evaluating argument list element number 1
error: evaluating argument list element number 1
error: called from:
error: E:\Dev\Octave\apps\findnextstring.m at line 7, column 6
error: E:\Dev\Octave\apps\parsepriceguidestring.m at line 33, column 14
error: E:\Dev\Octave\apps\readpriceguide.m at line 52, column 92
error: E:\Dev\Octave\apps\parse_wanted_list_and_price_guide.m at line 80, column 58
error: E:\Dev\Octave\apps\RUN_ALL.m at line 13, column 1
>>
Thanks!
I have an Intel i7, so I think I can use 8 cores, or only 4?
But now Bricklink is down, so I can't test your suggestions.
Thnks!
Checking 724 combinations of 1 vendors for those that have desired quantities...
Checking 261726 combinations of 2 vendors for those that have desired quantities...
Checking 62988724 combinations of 3 vendors for those that have desired quantities...
I've run it a few times changing the maximum number of stores each time and still the same results.
@dustinmroberts Did the program run to completion when it created that text file output, and are you using the latest version (V5.zip)? In older versions, that output file would mean no solutions were found, so it had nothing to report. More recent versions explicitly say 'No combinations found' to avoid confusion. PM me with any replies and we'll figure it out.
I would be willing to pay someone $25 for the PaB bsx file. So all you entrepreneurs, listen up!
My other idea was contacting TLG and asking them for an excel file of the info, explaining we wanted it so we could consider choosing PAB as a vendor option.
I've checked PAB already, the item's could be extracted easily from the site, eg:
http://customization.lego.com/en-US/pab/service/getBricks.aspx?st=5&sv=allbricks&pn=0&ps=500&cat=US
and then query all brick details by the appropriate query, eg.:
http://customization.lego.com/en-US/pab/service/getBrick.aspx?itemid=4211050&st=5&sv=allbricks&pageNo=0&pageSize=10&cat=US
*Problems/issues encountered so far:
- This is not "_A_ shop", so price and currency is different from the "cat=US" vs "cat=DE" (and so on).
- meanwhile basic bricks match the BL's ID's, like
LEGO: Element ID: 4211050 - 1 1/2 M Connecting Bush
Design ID: 32002
Exact Color: DARK STONE GREY
would "translate" to -->
Dark Bluish Gray Technic, Pin 3/4
http://www.bricklink.com/catalogPG.asp?P=32002&colorID=85
but:
LEGO: Element ID: 4275585 - Mini Upper Part "No. 1197"
Design ID: 76382
Exact Color: WHITE
cannot be automatically translated to:
White Torso Town Prisoner Number 50380, Dark Bluish Gray Stripes, Buttons, Hairy Chest Pattern / White Arms / Yellow Hands
http://www.bricklink.com/catalogPG.asp?P=973px432c01&colorID=1
This applies to most of the minifig parts I guess...
- I would need a color translation table, which maps LEGO color names to BL color ID-s. This is also not a big task to make, but I haven't completed that yet. If anyone has this already, post it or pm me. Currently available PAB colors [at some I put the BL color ID]... Fill the gap :-)
#BL_color_ID color_family "exact color [with spaces]"
11 Black BLACK
? Blue BRIGHT BLUE
? Blue EARTH BLUE
? Blue MEDIUM BLUE
? Blue TR. BLUE
? Blue TR. LIGHT BLUE
? Green BRIGHT GREEN
? Green BRIGHT YELLOWISH GREEN
? Green DARK GREEN
? Green TR. FLUORE. GREEN
? Green TR. GREEN
? Grey DARK STONE GREY
? Grey MEDIUM STONE GREY
? Orange-brown BRIGHT ORANGE
? Orange-brown DARK ORANGE
? Orange-brown REDDISH BROWN
? Orange-brown TR. BROWN
? Purple BRIGHT PURPLE
? Purple LIGHT PURPLE
? Purple MEDIUM LILAC
? Red BRIGHT RED
? Red NEW DARK RED
? Red TR. FLUORE.REDD. ORANGE
? Red TR. RED
1 White WHITE
3 Yellow BRICK YELLOW
? Yellow BRIGHT YELLOW
2 Yellow SAND YELLOW
? Yellow TR. YELLOW
- Decide what to do with "unavailable" items. Eg. item exists, but no stock.
- Updates...
I did some work on it, beta results here:
http://maques.hu/lego/auto-pab/
pablist-us.txt has 1532 entries, most of them matched easily with BrickLink, though I added some manual item/group translation tables.
Color translation table is there too, could be some errors in both the item and the color table.
80 items left on "todo" list to be manually looked up on BL - 99% minifig parts.
Could look up other country prices (UK, DE, HU, PL ...)
Could look up BL prices [1/lots for US/EU etc sellers]
Could create csv or any different format of the result
I can write a script that will create a .bsx text file, putting the expected markups around the values you've pulled. I can guarantee to make it readable by readLEGOPAB.m, but it may take some extra work to get it to import to BrickStore if one wanted to visually verify its contents or manually make an edit.
Is pablist-us.txt the format of your final product, and is there any opportunity for revision? Also note that BL prices will not be necessary, nor will a .csv format. PM me and we can finalize everything. Thanks for the help.
Thanks for doing this!
So in case PAB item updates or for any other reason I shall create a new bsx then they would be corrected and wouldn't need to have modified by "end-user" :-)
I am planning to release another revision update soon which will include this fix and few others, but I think I will wait until the work of @maques is complete to ensure compatibility. The draft .bsx file he has already provided is a huge timesaver.
- qty 9999 for item on stock and qty 0 for N/A stock item
- removed "fixme/error" items - so no ~80 warnings with BrickStore
- All LEGO/PAB information in Remarks field
Current issues/todo:
- 1 unknown item stayed with BrickStore, but I don't yet had time to check why, as item 50861 in color 11 [black] exist.
[Catalog: Parts: Tire & Tread: 50861 - Tire 21mm D. x 6mm City Motorcycle]
- The missing/to be checked-and-fixed 86 items are in:
http://maques.hu/lego/auto-pab/bltranslate-todo.txt
even if people here can lookup and match 1-1 item and post details here (eg what LEGO PAB Element # is the matching BL item #), then it would be helpful
- The "undetermined" elements come from EXACT element matching, like 3069/3070, where it comes from LEGO/PAB Design ID being 3069/3070.
Because for 3069b/3070b the PAB Design ID-s 30070/30039. Since there is no way for me to find out wheter a LEGO 3069/3070 is without or with a groove, so type A or B, I can automatch group (or single item).
After I automatch 3068/3069/3070/2412, there are still around 16 other items stay in undetermined state, eg. with undetermined stud, hinge, handle, ends&attachments, back and hole types.
What to do with them?
Anything else?
Wouldn't it be good/useful/interesting to set up a "phantom" Bricklink Store with the PAB items, so could be updated and matched wanted lists against it?
Then there could be different ones for countries as the PAB USD/EUR/GBP/HUF/PLN/HUF etc prices are static...
Problem would be that these shops would come out all the time but you wouldn't be able to order as it would be just a reference shop...
So it would need BL administration + programming to make these hidden / available per request and also not to count in the ~1500x9999 =~ 15 million extra parts...
is with automatch groups 3068/3069/3070/2412 to "b" (with groove/lip)
Still don't understand the unknown motorcycle tire. BrickStore's database out of sync, or not/can't do lookup to BL?