Lee Hoseop

Notepad++๋ฅผ ์ด์šฉํ•œ No-code ์ „์ฒ˜๋ฆฌ!

๏ปฟ
๏ปฟ
์•ˆ๋…•ํ•˜์„ธ์š”. ์˜ค๋Š˜์€ Teachable NLP๋ฅผ ํ•™์Šต์‹œํ‚ค๊ธฐ ์œ„ํ•ด Notepad++๋กœ ํ…์ŠคํŠธ ํŒŒ์ผ์„ preprocessing(์ „์ฒ˜๋ฆฌ) ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์†Œ๊ฐœํ•ด๋“œ๋ฆฌ๋ ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค. ๋ณดํ†ต ํ…์ŠคํŠธ ํŒŒ์ผ์„ preprocessing์„ ํ•˜๊ธฐ ์œ„ํ•ด์„  ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์–ธ์–ด๋ฅผ ์ตํ˜€์•ผ ํ•˜๊ณ  ๋ณต์žกํ•œ ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•ด์•ผ ํ•˜์ง€๋งŒ, ๊ทธ๋Ÿฐ ๊ฒƒ๋“ค ํ•„์š” ์—†์ด Notepad++๋ฅผ ์ด์šฉํ•ด preprosessing ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ๋ ค๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค!

ย Notepad++๋Š” Widows ์šด์˜์ฒด์ œ์—์„œ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ํ”„๋ฆฌ์›จ์–ด์ž…๋‹ˆ๋‹ค. ์šฉ๋Ÿ‰๋„ 10mb ์ •๋„๋กœ ์ž‘๊ณ  ๊ฐ€๋ฒผ์šฐ๋ฉฐ ๋น ๋ฅด๊ธฐ๊นŒ์ง€ ํ•ฉ๋‹ˆ๋‹ค. ๋งŒ์•ฝ, Linux ์šด์˜์ฒด์ œ๋ผ๋ฉด Notepadqq๋ฅผ ์‚ฌ์šฉํ•˜์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
Notepad++๋Š” ๋ฉ”๋ชจ์žฅ(Notepad)๋ฅผ ๊ฐ•ํ™” ํŒ์ด๋ž€ ์ด๋ฆ„์—์„œ ++ ๋ผ๋Š” ์ด๋ฆ„์„ ๊ฐ€์ง€๊ฒŒ ๋˜์—ˆ์œผ๋ฉฐ, ๊ทธ ์ด๋ฆ„์— ๊ฑธ๋งž๊ฒŒ ๋‹ค์–‘ํ•œ ๊ธฐ๋Šฅ๋“ค์„ ์ง€์›ํ•ฉ๋‹ˆ๋‹ค. ๊ทธ๋Ÿผ ์ด ๋‹ค์–‘ํ•œ ๊ธฐ๋Šฅ๋“ค์„ ์ด์šฉํ•ด ํ…์ŠคํŠธ ํŒŒ์ผ์„ preprocessing ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ Kaggle์—์„œ ๊ฐ€์ ธ์˜จ ๋ฐ์ดํ„ฐ๋ฅผ ์‚ฌ์šฉํ•ด ์†Œ๊ฐœํ•ด๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค.



Easy-1. ๋‹จ์–ด ๋‹ค๋ฃจ๊ธฐ


ย ๋ฌธ์„œ ํ”„๋กœ๊ทธ๋žจ์—์„œ ๋น ์งˆ ์ˆ˜ ์—†๋Š” ๋‹จ์–ด๋ฅผ ๋‹ค๋ฃจ๋Š” ์žˆ๋Š” ๊ธฐ๋Šฅ์ž…๋‹ˆ๋‹ค. Notepad++์—์„œ๋„ ๋‹จ์–ด๋“ค์„ ์ฐพ๊ณ  ๋ฐ”๊พธ๊ณ  ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
ย ์ƒ๋‹จ ๋ฐ”์˜ ๋ฐ”๊พธ๊ธฐ ๋ฒ„ํŠผ ๋˜๋Š” Ctrl + H ๋ฅผ ๋ˆ„๋ฅด๋ฉด ์œ„์™€ ๊ฐ™์€ ๋ฐ”๊พธ๊ธฐ ์ฐฝ์ด ๋‚˜์˜ค๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ ์ฐพ์„ ๋‚ด์šฉ๊ณผ ๋ฐ”๊ฟ€ ๋‚ด์šฉ์„ ์ž…๋ ฅํ•˜์—ฌ ๋‹จ์–ด๋“ค์„ ๋งˆ์Œ๋Œ€๋กœ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, ๋‹จ์–ด๊ฐ€ ์ž˜๋ชป ๋ฐ”๋€Œ๋Š” ๊ฒฝ์šฐ๋ฅผ ๋ฐฉ์ง€ํ•˜๊ธฐ ์œ„ํ•œ ๋‹จ์–ด ์™„์ „ ์ผ์น˜ ์˜ต์…˜๊ณผ ๋Œ€์†Œ๋ฌธ์ž ๊ตฌ๋ถ„ ์˜ต์…˜์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ๋‘ ์˜ต์…˜์„ ์ฒดํฌํ•˜์ง€ ์•Š์•˜์„ ๊ฒฝ์šฐ isolation์ด wasolation์œผ๋กœ ๋ณ€ํ•˜๊ณ  Is๋„ was๊ฐ€ ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ๋‹จ์–ด ์™„์ „ ์ผ์น˜๋ฅผ ์ฒดํฌํ•  ๊ฒฝ์šฐ isolation์€ ๋ณ€ํ•˜์ง€ ์•Š๊ณ  ๋Œ€์†Œ๋ฌธ์ž ๊ตฌ๋ถ„์„ ์ฒดํฌํ•˜๋ฉด Is๊ฐ€ ๋ณ€ํ•˜์ง€ ์•Š๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.


Easy-2. ๊ณต๋ฐฑ ๋ผ์ธ ์ œ๊ฑฐ

๏ปฟ
๏ปฟ ์–ด๋–ค ๋ฐ์ดํ„ฐ๋“ค์€ ์—ด์–ด๋ณด๋ฉด ํ•„์š” ์—†๋Š” ์•„๋ฌด๊ฒƒ๋„ ์—†๋Š” ๊ณต๋ฐฑ ๋ผ์ธ์ด ๋นˆ๋ฒˆํ•˜๊ฒŒ ์žˆ๋Š” ๋ฐ์ดํ„ฐ๋“ค์ด ์žˆ์Šต๋‹ˆ๋‹ค. ์šฐ๋ฆฌ๋“ค์€ ๊ณต๋ฐฑ ๋ผ์ธ์„ ํ•œ ๋ฒˆ์— ์•Œ์•„๋ณผ ์ˆ˜ ์žˆ์ง€๋งŒ, ๋ชจ๋ธ์„ ํ•™์Šต์‹œํ‚ฌ ๊ฒฝ์šฐ ์ด๋ฅผ ๊ตฌ๋ถ„ํ•ด๋‚ด์ง€ ๋ชปํ•ด ๊ทธ๋Œ€๋กœ ํ•™์Šตํ•ด๋ฒ„๋ ค ๊ฒฐ๊ณผ๋ฅผ ์–ด์ƒ‰ํ•˜๊ฒŒ ๋งŒ๋“ค์–ด ๋ฒ„๋ฆด ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
ย Notepad++์—์„œ๋Š” ์ด๋ฅผ ์‰ฝ๊ฒŒ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ƒ๋‹จ ๋ฐ”์˜ ํŽธ์ง‘ > ์ค„ ๋‹จ์œ„ ํŽธ์ง‘ > ๋นˆ ์ค„ ์ œ๊ฑฐ (๊ณต๋ฐฑ ๋ฌธ์ž ํฌํ•จ) ์„ ์„ ํƒํ•ด ๊ณต๋ฐฑ ๋ผ์ธ๋“ค์„ ์ œ๊ฑฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.ย 


Easy-3. ๋ฌธ๋‹จ ํ•ฉ์น˜๊ธฐ

๏ปฟ
๏ปฟ
ย  ์†Œ์„ค ๊ฐ™์€ ๋งŽ์€ ๊ธ€์ด ์žˆ๋Š” ๋ฐ์ดํ„ฐ๋“ค์€ ๊ธ€์„ ํ•œ๋ˆˆ์— ๋ณด๊ธฐ ์‰ฝ๊ฒŒ ํ•˜๊ธฐ ์œ„ํ•ด ๋ฌธ๋‹จ์˜ ๋์ด ์•„๋‹˜์—๋„ ์ค„๋ฐ”๊ฟˆ์„ ์‚ฝ์ž…ํ•ด ๋„ฃ์€ ๊ฒฝ์šฐ๋“ค์ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋Ÿฐ ์ค„๋ฐ”๊ฟˆ์„ ๊ทธ๋Œ€๋กœ ํ•™์Šต์‹œํ‚ค๋ฉด ์™„์„ฑ๋œ ๋ชจ๋ธ์—์„œ๋„ ๊ธ€์ด ๋‚˜์˜ค๋Š” ์ค‘๊ฐ„์ค‘๊ฐ„์— ์ค„๋ฐ”๊ฟˆ์ด ์‚ฝ์ž…๋˜์–ด ์ด์ƒํ•œ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜ํƒ€๋‚˜๊ฒŒ ๋  ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์Šต๋‹ˆ๋‹ค. ๋งŒ์•ฝ, ๋…ธ๋ž˜ ๊ฐ™์€ ๋ชจ๋ธ์„ ๋งŒ๋“œ๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ๋ฉด ํ•˜๋‚˜์˜ ๋ฌธ๋‹จ์œผ๋กœ ํ•ฉ์น˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
ย ๋จผ์ € ๋ฌถ์„ ์ค„๋“ค์„ ํ•˜๋‚˜๋กœ ๋“œ๋ž˜๊ทธํ•œ ๋’ค, ํŽธ์ง‘ > ์ค„๋‹จ์œ„ ํŽธ์ง‘ > ์ค„ ํ•ฉ์น˜๊ธฐ ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด ์ค„๋“ค์„ ํ•˜๋‚˜์˜ ๋ฌธ๋‹จ์œผ๋กœ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.


Easy-4. ์ค„ ์•ž๋’ค ๊ณต๋ฐฑ ์ œ๊ฑฐ

๏ปฟ
๏ปฟ ์–ด๋–ค ๋ฐ์ดํ„ฐ๋“ค์€ ๊ธ€์„ ๋ณด๊ธฐ ์ข‹๊ฒŒ ํ•˜๊ธฐ ์œ„ํ•ด ๋˜๋Š” ๋ฐ์ดํ„ฐ ํŠน์„ฑ์ƒ ์ค„์˜ ์•ž๋’ค๋กœ ๊ณต๋ฐฑ๋“ค์ด ์‚ฝ์ž…๋˜์–ด ์žˆ๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. GPT-2 ๋ชจ๋ธ ๊ฐ™์€ ๋ชจ๋ธ์„ ํ•™์Šต์‹œํ‚ฌ ๊ฒฝ์šฐ ์ด๋Ÿฐ ์ž˜๋ชป๋œ ๊ณต๋ฐฑ์„ ํ•™์Šตํ•˜๋ฉด ์ƒ์„ฑ๋œ ๊ฒฐ๊ณผ์— ์–ด์ƒ‰ํ•œ ๊ณต๋ฐฑ์ด ๋‚˜ํƒ€๋‚  ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ์ด๋Ÿฐ ๊ณต๋ฐฑ๋“ค์€ ์ง€์›Œ์ฃผ๋Š” ๊ฒƒ์ด ์ข‹์€๋ฐ, Notepad++๋Š” ์‰ฝ๊ฒŒ ์ œ๊ฑฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.ย 

ย ์ƒ๋‹จ ๋ฐ”์˜ ํŽธ์ง‘ > ๊ณต๋ฐฑ๊ธฐ๋Šฅ > ์ค„ ์‹œ์ž‘/๋ ๊ณต๋ฐฑ ์ œ๊ฑฐ ๊ธฐ๋Šฅ์œผ๋กœ ์†์‰ฝ๊ฒŒ ์ค„ ์•ž๋’ค์— ์žˆ๋Š” ๊ณต๋ฐฑ๋“ค์„ ์ œ๊ฑฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.


Easy-4. ํŠน์ • ๋ผ์ธ ์ œ๊ฑฐ


ํ…์ŠคํŠธ ํŒŒ์ผ์„ ์ „์ฒ˜๋ฆฌ ํ•˜๋‹ค ๋ณด๋ฉด, ํŠน์ • ๋ฌธ์ž๋ฅผ ๊ฐ€์ง„ ๋ผ์ธ์„ ์ œ๊ฑฐํ•˜๊ณ  ์‹ถ์€ ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿด ๋•Œ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ ๋จผ์ €, ์ €๋Š” ์ด ๊ธ€์—์„œ was๊ฐ€ ํฌํ•จ๋œ ์ค„๋“ค์„ ์ œ๊ฑฐํ•˜๊ณ  ์‹ถ์Šต๋‹ˆ๋‹ค. Ctrl + F ๋ฅผ ๋ˆŒ๋Ÿฌ ์ฑ…๊ฐˆํ”ผ ํƒญ์œผ๋กœ ๋„˜์–ด๊ฐ‘๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ์ฐพ์„ ๋‚ด์šฉ์„ ์ž…๋ ฅํ•˜๊ณ  ์ค„์„ ์ฑ…๊ฐˆํ”ผ ์˜ต์…˜์„ ์ฒดํฌํ•œ ๋’ค, ๋ชจ๋‘ ์ฐพ๊ธฐ๋ฅผ ๋ˆ„๋ฅด๋ฉด was๊ฐ€ ํฌํ•จ๋œ ์ค„์— O ํ‘œ์‹œ๊ฐ€ ์ƒ๊น๋‹ˆ๋‹ค.
์ƒ๋‹จ ๋ฐ”์˜ ์ฐพ๊ธฐ > ์ฑ…๊ฐˆํ”ผ > ์ฑ…๊ฐˆํ”ผ ๋œ ์ค„ ์‚ญ์ œํ•˜๊ธฐ ๋ฅผ ์„ ํƒํ•˜๋ฉด O ํ‘œ์‹œ๊ฐ€ ์žˆ๋˜ ์ค„๋“ค์ด ๋ชจ๋‘ ์‚ฌ๋ผ์ง€๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ๋งŒ์•ฝ ์ฑ…๊ฐˆํ”ผ ์•ˆ๋œ ์ค„ ์‚ญ์ œํ•˜๊ธฐ ๋ฅผ ์„ ํƒํ•˜๋ฉด ๋ฐ˜๋Œ€๋กœ ์ฑ…๊ฐˆํ”ผ๊ฐ€ ๋˜์–ด์žˆ์ง€ ์•Š์€ ์ค„๋“ค์ด ์‚ฌ๋ผ์ง€๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.


Hard-1. ์ •๊ทœ์‹


์—ฌ๊ธฐ์„œ ๋ถ€ํ„ฐ๋Š” ์กฐ๊ธˆ ์–ด๋ ค์šด ๋‚ด์šฉ์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ •๊ทœ์‹, ์ •๊ทœ ํ‘œํ˜„์‹์€ ํŠน์ • ๋ฌธ์ž๋ฅผ ๊ฒ€์ƒ‰ํ•˜๊ณ  ์ฒ˜๋ฆฌํ•˜๊ธฐ ์œ„ํ•œ ํŒจํ„ด์„ ๋œปํ•˜๋Š”๋ฐ ๊นŠ๊ฒŒ ์•Œ๊ณ  ์‹ถ์œผ์‹œ๋‹ค๋ฉด ์ด๊ณณ์„ ์ฐธ๊ณ ํ•˜์‹œ๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค. ํŒจํ„ด์„ ์งœ๋Š” ๊ฒƒ์ด ์–ด๋ ค์šธ ๋ฟ์ด์ง€ ๊ธฐ์กด ๋‹จ์–ด ๋‹ค๋ฃจ๊ธฐ๋‚˜ ์ฑ…๊ฐˆํ”ผ์ฒ˜๋Ÿผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ๊ฐ„๋‹จํ•œ ์‚ฌ์šฉ ์˜ˆ์‹œ๋ถ€ํ„ฐ ์•Œ๋ ค ๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค. ์ „์ฒ˜๋ฆฌ๋ฅผ ํ•˜๋‹ค ๋ณด๋ฉด ํŠน์ • ํŒจํ„ด์„ ๊ฐ€์ง„ ๋‹จ์–ด๋“ค์„ ์‚ญ์ œํ•˜๊ณ  ์‹ถ์„ ๋•Œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฐ ๊ฒƒ์—๋Š” Chapter ํ‘œ๊ธฐ, URL, HTML ํƒœ๊ทธ ๊ฐ™์€ ๊ฒƒ๋“ค์ด ์žˆ์ฃ .
์œ„ ์‚ฌ์ง„๋„ ๊ธ€์ž ์ค‘๊ฐ„์ค‘๊ฐ„์— URL ๋“ค์ด ์‚ฝ์ž…๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ด URL๋“ค์€ ์™„์ „ํžˆ ๊ฐ™์€ ๋ชจ์Šต์€ ์•„๋‹ˆ์ง€๋งŒ, ๊ฐ™์€ ํŒจํ„ด์„ ๊ฐ€์ง€๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ ‡์Šต๋‹ˆ๋‹ค, ๋ฐ”๋กœ ์ •๊ทœ์‹์„ ์‚ฌ์šฉํ•  ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค.
Ctrl + H ๋ฅผ ๋ˆŒ๋Ÿฌ ๋ฐ”๊พธ๊ธฐ ์ฐฝ์„ ์ผœ์ค๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ์™ผ์ชฝ ์•„๋ž˜ ์ฐพ๊ธฐ ๋ชจ๋“œ์—์„œ ์ •๊ทœ ํ‘œํ˜„์‹์„ ์ฒดํฌํ•˜์—ฌ ์ •๊ทœ์‹ ๋ชจ๋“œ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ๊ธฐ์กด ๋ฐ”๊พธ๊ธฐ์ฒ˜๋Ÿผ ์ฐพ์„ ๋‚ด์šฉ์— ์ •๊ทœ์‹์„ ๋„ฃ๊ณ  ๋ฐ”๊ฟ€ ๋‚ด์šฉ์„ ์ฑ„์šฐ๋ฉด ์ •๊ทœ์‹์— ๋งค์นญ๋˜๋Š” ๋‹จ์–ด๋“ค์„ ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋Ÿผ ์ด ์ •๊ทœ์‹์€ ์–ด๋–ป๊ฒŒ ๋งŒ๋“ค์ง€?๋ผ๋Š” ์˜๋ฌธ์ด ์ƒ๊ธธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋ช‡ ๊ฐ€์ง€ ๋ฌธ์ž์™€ ๊ทธ ์—ญํ• ์„ ์•Œ๋ฉด ์‰ฝ๊ฒŒ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, ํ•œ ๋ฒˆ ์ตํ˜€๋‘๋ฉด ๋‹ค๋ฅธ ๋ฌธ์„œ ํˆด์ด๋‚˜ ํ”„๋กœ๊ทธ๋ž˜๋ฐ ์–ธ์–ด์˜ ์ •๊ทœ์‹์ด ๋Œ€๋ถ€๋ถ„ ๋น„์Šทํ•˜๊ธฐ ๋•Œ๋ฌธ์— ์—ฌ๋Ÿฌ ๊ณณ์— ์‘์šฉํ•˜์‹ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋งŽ์€ ์ •๊ทœ์‹ ๋ฌธ์ž๋“ค์ด ์žˆ์ง€๋งŒ, ๋ช‡ ๊ฐ€์ง€๋งŒ ์„ค๋ช…ํ•ด ๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค. ๋” ๋งŽ์€ ๋ฌธ์ž๋“ค์€ ์ด๊ณณ์„ ์ฐธ๊ณ ํ•ด๋ณด์‹œ๋ฉด ์ข‹์„ ๊ฒ๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ์ œ๊ฐ€ ์ ์€ ๊ฒƒ์€ ๋งŽ์€ ๋ฌธ์ž๋“ค ์ค‘ ์ผ๋ถ€๋ถ„์ผ ๋ฟ, ๋” ๋งŽ์€ ์ •๊ทœ์‹ ๋ฌธ์ž๋“ค์ด ์กด์žฌํ•ฉ๋‹ˆ๋‹ค.ย 


Hard-2. ๋งคํฌ๋กœ


๋งคํฌ๋กœ๋Š” ์—ฌ๋Ÿฌ ํ–‰๋™์„ ํ•˜๋‚˜๋กœ ๋ฌถ๋Š” ๊ฒƒ์„ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. Notepad++์—์„  ํ‚ค ์ž…๋ ฅ๋“ค์„ ํ•˜๋‚˜์˜ ๋งคํฌ๋กœ๋กœ ๋งŒ๋“ค์–ด ๋ฐ˜๋ณต์‹œํ‚ค๋Š” ๊ฒƒ์ด ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ๋งŒ์•ฝ, ์ •๊ทœ์‹๊ณผ ์ฑ…๊ฐˆํ”ผ๋กœ ํ•ด๊ฒฐ ๋ชปํ•˜์ง€๋งŒ ์ผ์ • ํŒจํ„ด์„ ๊ฐ€์กŒ๋‹ค๋ฉด ์ด ๋ฐฉ๋ฒ•์„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ
์ €๋Š” ์—ฌ๊ธฐ์„œ was๋“ค์„ is๋กœ ๋ฐ”๊พธ๊ณ  ์‹ถ์Šต๋‹ˆ๋‹ค. ๋ฐ”๊พธ๊ธฐ๋ฅผ ์ด์šฉํ•ด ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์ง€๋งŒ, ๋งคํฌ๋กœ๋ฅผ ์ด์šฉํ•ด ๋ฐ”๊พธ๋Š” ๋ฐฉ๋ฒ•์„ ์•Œ๋ ค๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ ๋จผ์ € Ctrl + F ๋ฅผ ๋ˆŒ๋Ÿฌ ์ฐพ๊ธฐ ์ฐฝ์œผ๋กœ ๋“ค์–ด๊ฐ€ ์ฐพ์„ ๋‚ด์šฉ์— was๋ฅผ ์ ์–ด์ค๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  ์ฐพ๊ธฐ ์ฐฝ์„ ๋„๊ณ  F3์„ ๋ˆŒ๋Ÿฌ was๋“ค์ด ์ฐพ์•„์ง€๋Š”์ง€ ํ™•์ธํ•ด๋ด…๋‹ˆ๋‹ค.

์ปค์„œ๋ฅผ ๊ธ€์˜ ๋งจ ์•ž์— ์œ„์น˜ ์‹œ์ผœ์ค€ ๋’ค, ์ƒ๋‹จ ๋ฐ”์˜ ๋งคํฌ๋กœ > ๊ธฐ๋ก ์‹œ์ž‘์„ ๋ˆŒ๋Ÿฌ์ค๋‹ˆ๋‹ค. ๊ทธ๋ฆฌ๊ณ  F3 > Del > is ํƒ€์ดํ•‘ ์„ ํ•œ๋ฒˆ ํ•ด์ฃผ๊ณ  ๋งคํฌ๋กœ > ๊ธฐ๋ก ์ค‘์ง€ ๋ฅผ ๋ˆŒ๋Ÿฌ์ค๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋ฉด F3 > Del > is ํƒ€์ดํ•‘์ด ํ•˜๋‚˜์˜ ๋งคํฌ๋กœ๋กœ ์ €์žฅ๋˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.
๋งคํฌ๋กœ > ์‹คํ–‰ ์„ ๋ˆ„๋ฅด๊ฑฐ๋‚˜ Ctrl + Shift + P ๋ฅผ ๋ˆŒ๋Ÿฌ ์•ž์„œ ํ–ˆ๋˜ ์ผ์ด ๋ฐ˜๋ณต ๋˜๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.

์ •์ƒ ์ž‘๋™์ด ๋˜๋Š”์ง€ ํ™•์ธํ–ˆ๋‹ค๋ฉด, ๋งคํฌ๋กœ > ๋งคํฌ๋กœ ๋ฐ˜๋ณต ์‹คํ–‰ ์„ ๋ˆŒ๋Ÿฌ ํŒŒ์ผ ๋๊นŒ์ง€ ์‹คํ–‰์„ ์ฒดํฌํ•˜๊ณ  ์‹คํ–‰์„ ๋ˆŒ๋Ÿฌ์ค๋‹ˆ๋‹ค. ๋งŒ์•ฝ ์ผ์ • ํšŸ์ˆ˜๋งŒ ์‹คํ–‰๋˜๊ธธ ์›ํ•˜๋ฉด ์‹คํ–‰์„ ์ฒดํฌํ•˜๊ณ  ์›ํ•˜๋Š” ํšŸ์ˆ˜๋ฅผ ๊ธฐ์ž…ํ•˜๊ณ  ์‹คํ–‰์„ ๋ˆŒ๋Ÿฌ์ฃผ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ๊ทธ๋Ÿผ ์งœ์ž”! ๋ฌธ์„œ์˜ was๋“ค์ด ๋ชจ๋‘ is๋กœ ๋ฐ”๋€Œ์—ˆ์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ, ๋งˆ์ง€๋ง‰ was๊ฐ€ isis๊ฐ€ ๋˜์–ด๋ฒ„๋ ธ์Šต๋‹ˆ๋‹ค. ์ด๊ฑด 22๋ฒˆ์งธ ์ค„๊นŒ์ง€ ์‹คํ–‰๋ผ์•ผ ํ•˜๋Š”๋ฐ 22๋ฒˆ์งธ ์ค„์— was๊ฐ€ ์—†์–ด ๋งˆ์ง€๋ง‰ was์—์„œ ๋งคํฌ๋กœ๊ฐ€ ํ•œ ๋ฒˆ ๋” ์‹คํ–‰๋œ ๊ฒฐ๊ณผ์ž…๋‹ˆ๋‹ค. ์ด๋ฅผ ๋ฐฉ์ง€ํ•˜๋ ค๋ฉด, ์ฐพ๊ธฐ ์ฐฝ์—์„œ ์ผ์น˜ํ•˜๋Š” ๊ฐœ์ˆ˜๋ฅผ ๋ˆ„๋ฅด์‹œ๋ฉด was๊ฐ€ ๋ช‡ ๊ฐœ ์žˆ๋Š”์ง€ ํ‘œ์‹œ๋˜์–ด ๋งคํฌ๋กœ ๋ฐ˜๋ณต ์‹คํ–‰์—์„œ ๊ทธ ๊ฐœ์ˆ˜๋งŒํผ ๋ฐ˜๋ณต ์‹œํ‚ค๋ฉด ๋ฉ๋‹ˆ๋‹ค.

๋งˆ์ง€๋ง‰์œผ๋กœ, ๋งคํฌ๋กœ๋ฅผ ๋‹ค์Œ์— ๋˜ ์‚ฌ์šฉํ•˜๊ธธ ์›ํ•˜๋ฉด ๋งคํฌ๋กœ > ํ˜„์žฌ ๊ธฐ๋ก๋œ ๋งคํฌ๋กœ ์ €์žฅ ์„ ๋ˆ„๋ฅด๊ณ  ์›ํ•˜๋Š” ๋‹จ์ถ•ํ‚ค๋ฅผ ์ง€์ •ํ•˜๋ฉด ๋งคํฌ๋กœ๊ฐ€ ์ง€์ •๋˜์–ด ๋‹ค์Œ์—๋„ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.


Start training!

๏ปฟ
๏ปฟ์ง€๊ธˆ๊นŒ์ง€ Notepad++์ด์šฉํ•ด ์ฝ”๋“œ ์—†์ด ํ…์ŠคํŠธ ํŒŒ์ผ์„ ์ „์ฒ˜๋ฆฌ ํ•˜๋Š” ๋ฒ•์— ๋Œ€ํ•œ ์†Œ๊ฐœ์˜€์Šต๋‹ˆ๋‹ค. ์ด ๋ฐฉ๋ฒ•๋“ค์„ ์ด์šฉํ•˜์—ฌ ์—ฌ๋Ÿฌ๋ถ„์˜ ๋ฐ์ดํ„ฐ๋ฅผ ๋” ๋ฉ‹์ง€๊ฒŒ ๋งŒ๋“œ์…จ์„ ๊ฒƒ์ด๋ผ ์ƒ๊ฐํ•ฉ๋‹ˆ๋‹ค! ์ด์ œ Teachable NLP ์—์„œ ์™„์„ฑ๋œ ๋ฐ์ดํ„ฐ๋ฅผ ์ด์šฉํ•ด ์—ฌ๋Ÿฌ๋ถ„ ๋งŒ์˜ ๋ชจ๋ธ์„ ๋งŒ๋“ค์–ด๋ณผ ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค!
Like Comment

New York Times๋กœ ๋งŒ๋“ค์–ด๋‚ธ ๊ฐ€์งœ ๋‰ด์Šค ์ƒ์„ฑ๊ธฐ ๐Ÿ˜ฑ

๏ปฟ
๏ปฟTeachable NLP: link
Tabtab: link
API page: link

์ฝ”๋“œ์™€ GPU ์ž์› ์—†์ด GPT-2๋ชจ๋ธ์„ ํŠœ๋‹ํ•  ์ˆ˜ ์žˆ๋Š”! Teachable NLP๊ฐ€ ์ •์‹ ์ถœ์‹œ๋˜์—ˆ์Šต๋‹ˆ๋‹ค! ๐Ÿ‘๐Ÿ‘๐Ÿ‘
์˜ค๋Š˜์€ GPT-2๋ฅผ ํ•™์Šต์‹œํ‚ค๊ธฐ ์œ„ํ•ด ๋‰ด์š• ํƒ€์ž„์ฆˆ์˜ ๋ฐ์ดํ„ฐ๋ฅผ ์ค€๋น„ํ–ˆ์Šต๋‹ˆ๋‹ค. GPT ๋ชจ๋ธ์ด ๊ณต๊ฐœ๋˜์—ˆ์„ ๋•Œ, ์‚ฌ๋žŒ๋“ค์€ ๋ฌด์ˆ˜ํžˆ ๋งŽ์€ ๊ฐ€์งœ ๋‰ด์Šค๋“ค์ด ์ƒ์„ฑ๋  ๊ฒƒ์„ ๋‘๋ ค์›Œํ–ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ์ด ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์™”๋Š”๋ฐ, ๊ณผ์—ฐ ์ด๋ฒˆ์— ์ƒ์„ฑ๋œ ๋ชจ๋ธ์ด ๊ทธ ๋ช…์„ฑ์ฒ˜๋Ÿผ ๋ฌด์‹œ๋ฌด์‹œํ•œ ๊ฐ€์งœ ๋‰ด์Šค๋ฅผ ๋งŒ๋“ค์–ด ๋‚ผ๊นŒ์š”? ์˜ค๋Š˜ ํ•œ๋ฒˆ ์•Œ์•„๋ณด๋„๋ก ํ•ฉ์‹œ๋‹ค!


ย 1. Data

๏ปฟ
๏ปฟ
URL: http://www.nytimes.com/2016/06/30/sports/baseball/washington-nationals-max-scherzer-baffles-mets-completing-a-sweep.html

WASHINGTON โ€” Stellar pitching kept the Mets afloat in the first half of last season despite their offensive woes. But they cannot produce an encore of their pennant-winning season if their lineup keeps floundering while their pitching is nicked, bruised and stretched thin.
โ€œWe were going to ride our pitching,โ€ Manager Terry Collins said before Wednesdayโ€™s game. โ€œBut weโ€™re not riding it right now. Weโ€™ve got as many problems with our pitching as we do anything.โ€
Wednesdayโ€™s 4-2 loss to the Washington Nationals was cruel for the already-limping Mets. Pitching in Steven Matzโ€™s place, the spot starter Logan Verrett allowed two runs over five innings. But even that was too large a deficit for the Metsโ€™ lineup to overcome against Max Scherzer, the Nationalsโ€™ starter.
โ€œWeโ€™re not even giving ourselves chances,โ€ Collins said, adding later, โ€œWe just canโ€™t give our pitchers any room to work.โ€
The Mets did not score until the ninth inning, when a last-gasp two-run homer by James Loney off Nationals reliever Shawn Kelley snapped a streak of 23 scoreless innings for the team.
...

์˜ค๋Š˜ ์ค€๋น„ํ•œ ๋‰ด์š•ํƒ€์ž„์ฆˆ ๊ธฐ์‚ฌ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. Kaggle์—์„œ ๊ฐ€์ ธ์™”์œผ๋ฉฐ CC0: Public Domain ๋ผ์ด์„ผ์Šค๋กœ ์ž์œ ๋กญ๊ฒŒ ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ํฌ๊ธฐ๋Š” ๋ฌด๋ ค 43mb์˜ ์—„์ฒญ๋‚œ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค!!! ๊ฒŒ๋‹ค๊ฐ€ ๋ฐ”๋กœ ํ•™์Šต์— ์‚ฌ์šฉํ•ด๋„ ๋  ๋งŒํผ ๊น”๋”ํ•œ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค๋งŒ, ๊ธฐ์‚ฌ์˜ URL๋“ค์ด ์žˆ์œผ๋‹ˆ ๊ทธ๊ฒƒ๋งŒ ์ง€์šฐ๋ฉด ๋  ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.


2-1. ์ „์ฒ˜๋ฆฌ with No-code!

ย 
ย ์ฝ”๋“œ ์—†์ด ์ „์ฒ˜๋ฆฌ ํ•˜๋Š” ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค! ๋ฐ”๋กœ Notepad++๋ฅผ ์‚ฌ์šฉํ•œ ๋ฐฉ๋ฒ•์ž…๋‹ˆ๋‹ค. Window์™€ Linux์—์„œ ์‚ฌ์šฉ ๊ฐ€๋Šฅํ•œ ํ”„๋กœ๊ทธ๋žจ(!Mac์—์„  ์‚ฌ์šฉํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค!)์ธ Notepad++๋Š” ํ…์ŠคํŠธ ํŒŒ์ผ์„ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ๋Š” ๋‹ค์–‘ํ•œ ๊ธฐ๋Šฅ๋“ค์„ ์ œ๊ณตํ•˜๋Š”๋ฐ, ๊ทธ์ค‘ ๋ฐ”๊พธ๊ธฐ ๊ธฐ๋Šฅ์„ ์ด์šฉํ•ด ์ „์ฒ˜๋ฆฌํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ๋‹จ, ์ด ๋ฐฉ๋ฒ•์„ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด์„  Notepad++์˜ ์ •๊ทœ ํ‘œํ˜„์‹์„ ์•Œ์•„์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์ •๊ทœ ํ‘œํ˜„์‹์ด๋ž€ ํŠน์ • ๋ฌธ์ž๋ฅผ ๊ฒ€์ƒ‰ํ•˜๊ณ  ์ฒ˜๋ฆฌํ•˜๊ธฐ ์œ„ํ•œ ํŒจํ„ด์„ ๋œปํ•˜๋Š”๋ฐ ๊นŠ๊ฒŒ ์•Œ๊ณ  ์‹ถ์œผ์‹œ๋‹ค๋ฉด ์ด๊ณณ์„ ์ฐธ๊ณ ํ•˜์‹œ๋ฉด ์ข‹์„ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.ย 

ย  ๋จผ์ € Notepad++๋กœ ํ…์ŠคํŠธ ํŒŒ์ผ์„ ์‹คํ–‰ํ•œ ๋’ค, ์ฐพ๊ธฐ์˜ ๋ฐ”๊พธ๊ธฐ๋ฅผ ๋ˆŒ๋Ÿฌ์ค๋‹ˆ๋‹ค. ๋‹จ์ถ•ํ‚ค๋Š” Ctrl + H ์ž…๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ ์ฐพ์„ ๋‚ด์šฉ์— URL: [\w:./-]*.html ์„ ์ ์–ด์ค๋‹ˆ๋‹ค. ์ด ๋œป์„ ์„ค๋ช…ํ•ด๋“œ๋ฆฌ์ž๋ฉด, [\w:./-]์—์„œ \w๋Š” 0~9์˜ ์ˆซ์ž์™€ a~z, A~Z์˜ ๋ฌธ์ž๋ฅผ ์˜๋ฏธํ•ฉ๋‹ˆ๋‹ค. []๋Š” ๊ด„ํ˜ธ ์•ˆ์˜ ๋ฌธ์ž๋“ค ์ค‘ ํ•˜๋‚˜๋ผ๋Š” ๋œป์œผ๋กœ [\w:./-] ์€ \w๊ณผ ํŠน์ˆ˜๋ฌธ์ž :./- ์ค‘์— ํ•˜๋‚˜๋ผ๋Š” ๋œป์ด ๋ฉ๋‹ˆ๋‹ค. *์€ ๋ฐ”๋กœ ์•ž์˜ ๋ฌธ์ž๊ฐ€ 0๋ฒˆ ์ด์ƒ ๋ฐ˜๋ณต๋œ๋‹ค๋Š” ๋œป์ž…๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ, URL:๋กœ ์‹œ์ž‘ํ•˜๊ณ  ์ค‘๊ฐ„์— [\w:./-]์ด 0๋ฒˆ ์ด์ƒ ๋ฐ˜๋ณต๋˜๊ณ , .html ๋กœ ๋๋‚˜๋Š” ๋ชจ๋“  ๋ฌธ์ž๋“ค๊ณผ ๋งค์นญ ๋˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ๊ทธ๊ฒƒ๋งŒ์œผ๋กœ ๋๋‚ด๋ฉด ์•„์‰ฌ์šฐ๋‹ˆ ๋‰ด์Šค ๊ฐ„ ๊ตฌ๋ถ„๊ณผ ๋‰ด์Šค์˜ ์‹œ์ž‘์„ ์•Œ๋ ค์ฃผ๊ธฐ ์œ„ํ•ด ๋ฐ”๊ฟ€ ๋‚ด์šฉ์— <news>๋ฅผ ๋„ฃ์–ด์ฃผ๊ฒ ์Šต๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ์ง ! URL๋“ค์ด ๋ชจ๋‘ <news>๋กœ ๋ฐ”๋€Œ์—ˆ์Šต๋‹ˆ๋‹ค. ์ด์ œ <news> ์•ž๋’ค๋กœ ์žˆ๋Š” ๋นˆ ๋ผ์ธ๋“ค๋งŒ ์ œ๊ฑฐํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. ๋นˆ ๋ผ์ธ๋“ค์€ ์ •๊ทœ ํ‘œํ˜„์‹์„ ๋งŒ๋“ค์ง€ ์•Š์•„๋„ ์‰ฝ๊ฒŒ ์ง€์šธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹คย 
ย ์ƒ๋‹จ ๋ฐ”์˜ ํŽธ์ง‘ > ์ค„ ๋‹จ์œ„ ํŽธ์ง‘ > ๋นˆ ์ค„ ์ œ๊ฑฐ(๊ณต๋ฐฑ ๋ฌธ์ž ํฌํ•จ) ์„ ๋ˆŒ๋Ÿฌ์ค๋‹ˆ๋‹ค. ๊ทธ๋Ÿผ ์งœ์ž”! ๊ณต๋ฐฑ ๋ผ์ธ๋“ค์ด ๋ชจ๋‘ ์ œ๊ฑฐ๋˜์—ˆ์Šต๋‹ˆ๋‹ค! ์ด์ œ ๋ฐ”๋กœ ํ•™์Šต์— ์‚ฌ์šฉํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค!ย 

2-2.ย  ์ „์ฒ˜๋ฆฌ with Code!ย 


with open(source_dir + '/' + text_file, 'r', encoding='utf-8') as f:

    name = text_file.split('.')[0]

    with open(result_dir + '/' + name + '.txt', 'w', encoding='utf-8-sig') as r:
        lines = f.readlines()

        for line in lines:
			# ๊ณต๋ฐฑ์ผ ๊ฒฝ์šฐ ์ ์ง€์•Š๊ณ  ๊ฑด๋„ˆ ๋œ์ž…๋‹ˆ๋‹ค.
            if line == "\\n" or line == "" or line == " " or line == " \\n" or not line:
                continue
		
			# URL: ๋กœ ์‹œ์ž‘ํ•˜๋Š” ๋ผ์ธ์„ <news>๋กœ ๋ฐ”๊ฟ”์ค๋‹ˆ๋‹ค.
            if line.startswith('URL:'):
                r.write('<news>' + '\\n')
                continue

			# ํ˜น์‹œ ๋ชจ๋ฅด๋‹ˆ ๊ณต๋ฐฑ์„ ์ •๊ทœํ™”ํ•ด์ฃผ๋Š” ๋ถ€๋ถ„๋„ ์ถ”๊ฐ€ํ•ด์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค.
            text = " ".join(line.split())

            r.write(text + '\\n')
ย ๋‰ด์š• ํƒ€์ž„์ฆˆ ๋ฐ์ดํ„ฐ๋ฅผ ์ „์ฒ˜๋ฆฌํ•˜๋Š” ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค. URL ์ œ๊ฑฐ๋ฅผ ์ œ์™ธํ•˜๋ฉด ํ•ด์•ผ ํ•  ์ž‘์—…์ด ์—†๊ธฐ ๋•Œ๋ฌธ์— ๋งค์šฐ ์‹ฌํ”Œํ•ฉ๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ, ๊ทธ๊ฒƒ๋งŒ์œผ๋กœ ๋๋‚ด๋ฉด ์•„์‰ฌ์šฐ๋‹ˆ ๋‰ด์Šค ๊ฐ„ ๊ตฌ๋ถ„๊ณผ ๋‰ด์Šค์˜ ์‹œ์ž‘์„ ์•Œ๋ ค์ฃผ๊ธฐ ์œ„ํ•ด ๊ธฐ์กด URL์ด ์žˆ๋˜ ์ž๋ฆฌ์— <news>๋ฅผ ๋„ฃ์–ด์ฃผ๊ฒ ์Šต๋‹ˆ๋‹ค.ย 


<news>
WASHINGTON โ€” Stellar pitching kept the Mets afloat in the first half of last season despite their offensive woes. But they cannot produce an encore of their pennant-winning season if their lineup keeps floundering while their pitching is nicked, bruised and stretched thin.
โ€œWe were going to ride our pitching,โ€ Manager Terry Collins said before Wednesdayโ€™s game. โ€œBut weโ€™re not riding it right now. Weโ€™ve got as many problems with our pitching as we do anything.โ€
Wednesdayโ€™s 4-2 loss to the Washington Nationals was cruel for the already-limping Mets. Pitching in Steven Matzโ€™s place, the spot starter Logan Verrett allowed two runs over five innings. But even that was too large a deficit for the Metsโ€™ lineup to overcome against Max Scherzer, the Nationalsโ€™ starter.
โ€œWeโ€™re not even giving ourselves chances,โ€ Collins said, adding later, โ€œWe just canโ€™t give our pitchers any room to work.โ€
The Mets did not score until the ninth inning, when a last-gasp two-run homer by James Loney off Nationals reliever Shawn Kelley snapped a streak of 23 scoreless innings for the team.
...

ย ์‹ฌํ”Œํ•˜๊ฒŒ ์ „์ฒ˜๋ฆฌ๋œ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ๋ณต์žกํ•œ ๋กœ์ง์ด ์—†๊ธฐ ๋•Œ๋ฌธ์— ์ „์ฒ˜๋ฆฌ๊ฐ€ ๊ธˆ๋ฐฉ๊ธˆ๋ฐฉ ๋๋‚ฌ์Šต๋‹ˆ๋‹ค! ์–ด์งธ์„œ์ธ์ง€ ๋ฐ์ดํ„ฐ ์‚ฌ์ด์ฆˆ๊ฐ€ 43.9mb๋กœ ์ฆ๊ฐ€ํ–ˆ์ง€๋งŒ, ๋ฐ”๋กœ ํ•™์Šต์— ์‚ฌ์šฉํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค!ย 


3. Training


๏ปฟ
๏ปฟ ์ด๋ฒˆ์—” small ๋ชจ๋ธ๊ณผ 4 epoch๋ฅผ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค. ๋ณธ๋ž˜ ๋ฐ์ดํ„ฐ๊ฐ€ ํฌ๊ธฐ ๋•Œ๋ฌธ์— medium ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•˜๊ณ  ์‹ถ์—ˆ์ง€๋งŒ, ํ•™์Šต ์‹œ๊ฐ„์ด 7์‹œ๊ฐ„์ด๋‚˜ ๊ฑธ๋ฆฌ๊ธธ๋ž˜ small ๋ชจ๋ธ์„ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค.ย 

๏ปฟ
๏ปฟ Small ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•œ ๊ฒฐ๊ณผ 2์‹œ๊ฐ„ 30๋ถ„์ด ๊ฑธ๋ ธ์Šต๋‹ˆ๋‹ค. Medium์˜ 7์‹œ๊ฐ„๋ณด๋‹ค๋Š” ์ ์€ ์‹œ๊ฐ„์ด์ง€๋งŒ, ์ƒ๋‹นํžˆ ์˜ค๋žœ ์‹œ๊ฐ„์ด ๊ฑธ๋ ธ์Šต๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ Training์„ ๋Œ๋ ค๋†จ๋‹จ ๊ฑธ ์žŠ๊ณ  ์žˆ๋˜ ์‚ฌ์ด... Boom! ์ œ ๋ชจ๋ธ์ด ์™„์„ฑ๋˜์—ˆ๋‹ค๋Š” ๋ฉ”์ผ ์•Œ๋ฆผ์ด ์™”์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿผ ์ด์ œ ๋ชจ๋ธ์„ ํ…Œ์ŠคํŠธํ•ด๋ณผ ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค!ย 


4. Generate

๏ปฟ
๏ปฟ Test your model์„ ์‚ฌ์šฉํ•ด Tabtab์—์„œ ๋ชจ๋ธ์„ ํ…Œ์ŠคํŠธํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ๊ณผ์—ฐ ์‚ฌ๋žŒ๋“ค์ด ๋‘๋ ค์›Œํ–ˆ๋˜ ๊ฒƒ์ฒ˜๋Ÿผ ๋ฌด์‹œ๋ฌด์‹œํ•œ ๊ฐ€์งœ ๋‰ด์Šค๊ฐ€ ์ƒ์„ฑ๋ ๊นŒ์š”? ๋‘๊ทผ๋‘๊ทผํ•ฉ๋‹ˆ๋‹ค.ย 


<news>
PHILADELPHIA โ€” The Mets may not know for years whether David Wright will be their everyday third baseman, but sooner or later, he will.
Wright, who was suspended for 162 games without pay by Major League Baseball for violating its domestic violence policy last month, announced on Friday that he would sit out the first three weeks of the season while the team tried to convince him to return. A decision on his return is expected by the end of June. Wright was arrested Oct. 31 after his wife told the police that his former wife had told him that she had grabbed his throat and shoved him into a sliding-glass door at a Hawaiian resort, leaving him with severe rope burns. He returned a short time later on Aug. 9 and was charged with battery. But on Sunday, during a news conference at Major league Baseballโ€™s spring training complex in Arizona, Wright said he no longer believed that was where he had been arrested. It was unclear why he left the country or what role, if any, his lawyers hoped he might have played in inspiring others to act on their own. The police were not immediately available to comment. Messages left with the United States attorneyโ€œwere not returned.โ€

<news> ๋ฅผ ์ž…๋ ฅ์œผ๋กœ ๋งŒ๋“ค์–ด์ง„ ๋‰ด์š• ํƒ€์ž„์ฆˆ ๊ธฐ๋ฐ˜ ๊ฐ€์งœ ๋‰ด์Šค์ž…๋‹ˆ๋‹ค. ๋ฏธ๊ตญ ์•ผ๊ตฌํŒ€์˜ ์Šคํƒ€์ด์ž ์ „์„ค์ธ David Wright ์„ ์ˆ˜๋ฅผ ํ•œ์ˆœ๊ฐ„์— ๋‚˜์œ ์‚ฌ๋žŒ์œผ๋กœ ๋งŒ๋“ค์–ด๋ฒ„๋ฆฌ๋Š” ์ž”ํ˜นํ•œ ๋‰ด์Šค๊ฐ€ ๋‚˜์™”์Šต๋‹ˆ๋‹ค. ๋ชจ๋ธ์ด ์™„์„ฑ๋˜๊ธฐ ์ „๊นŒ์ง„ ์ •๋ง ๋ง์ด ๋˜๋Š” ๊ฐ€์งœ ๋‰ด์Šค๊ฐ€ ๋‚˜์˜ฌ๊นŒ ์˜์‹ฌํ–ˆ์—ˆ๋Š”๋ฐ, ํ•œ ์‚ฌ๋žŒ์˜ ๋ช…์„ฑ์„ ๋”๋Ÿฝํ˜€ ๋ฒ„๋ฆฌ๋Š” ๋ฌด์‹œ๋ฌด์‹œํ•œ ๋‰ด์Šค๊ฐ€ ๋‚˜์™€๋ฒ„๋ ธ์Šต๋‹ˆ๋‹ค.
์—ฌ๋Ÿฌ๋ถ„์€ ์ด ๊ฐ€์งœ ๋‰ด์Šค ์ƒ์„ฑ๊ธฐ์— ๋Œ€ํ•ด ์–ด๋–ป๊ฒŒ ์ƒ๊ฐํ•˜์‹œ๋‚˜์š”? ๋‘๋ ค์šฐ์‹ ๊ฐ€์š”? ํฅ๋ฏธ๋กœ์šฐ์‹ ๊ฐ€์š”? ๋น„๋ก ์ด๋ฒˆ์—” ๋ฌด์‹œ๋ฌด์‹œํ•œ ๋‰ด์Šค๋ฅผ ๋งŒ๋“ค์–ด๋ƒˆ์ง€๋งŒ, ๊ณผ์—ฐ ๋‹ค์Œ๋ฒˆ์—๋„ ๊ทธ๋Ÿด๊นŒ์š”?
Like Comment

Teachable NLP๋ฅผ ์ด์šฉํ•œ ๊ณตํฌ ์†Œ์„ค Lovecraft ๋ชจ๋ธ ๋งŒ๋“ค๊ธฐ

๏ปฟ
๏ปฟ
Teachable NLP: link
Tabtab: link
ย 
์•ˆ๋…•ํ•˜์„ธ์š” ์˜ค๋Š˜์€ ์ œ๊ฐ€ ์ด์ „์— ์‚ฌ์šฉํ•ด ๋ณธ ์ ์ด ์žˆ๋Š” ๋ฐ์ดํ„ฐ๋ฅผ ์ด์šฉํ•ด Teachable NLP GPT-2๋ฅผ fine-tuning ํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ๋Š” Kaggle์—์„œ ๊ฐ€์ ธ์˜จ ์œ ๋ช…ํ•œ ํ˜ธ๋Ÿฌ์†Œ์„ค์ธ LoveCraft์˜ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ์ด ๋ฐ์ดํ„ฐ ๋˜ํ•œ CC0: Public Domain ๋ฐ์ดํ„ฐ๋กœ ์ž์œ ๋กญ๊ฒŒ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!


1. ๋ฐ์ดํ„ฐ


A Reminiscence of Dr. Samuel Johnson

The Privilege of Reminiscence, however rambling or tiresome, is one generally allowโ€™d
to the very aged; indeed, โ€™tis frequently by means of such Recollections that the obscure
occurrences of History, and the lesser Anecdotes of the Great, are transmitted to Posterity.
Thoโ€™ many of my readers have at times observโ€™d and remarkโ€™d
a Sort of antique Flow in my Stile of Writing, it hath pleased me to pass amongst the Members
of this Generation as a young Man, giving out the Fiction that I was born in 1890, in America.
I am now, however, resolvโ€™d to unburthen myself of a Secret which I have hitherto kept
throโ€™ Dread of Incredulity; and to impart to the Publick a true knowledge of my long years,
...

ย ์ด 102๊ฐœ์˜ 7.58mb ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ, ๋ชจ๋“  ๋ฐ์ดํ„ฐ๋ฅผ ํ•œ ๊ณณ์— ๋ชจ์•„๋‘” concat.txt์„ ์ œ์™ธํ•˜๋ฉด 101์˜ 3.58mb์˜ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ๊ทธ๋ž˜๋„ ์ƒ๋‹นํžˆ ๋งŽ์€ ์–‘์˜ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค!!
ย ์ข‹์€ ๋ฐ์ดํ„ฐ์ง€๋งŒ, ์ œ๊ฐ€ ์›ํ•˜๋Š” ๋ชจ์Šต์€ ์•„๋‹ˆ๊ธฐ์— ํ•™์Šต์‹œํ‚ค๊ธฐ ์ „์— ์ „์ฒ˜๋ฆฌ๋ฅผ ํ•ด์ค๋‹ˆ๋‹ค.


2. ์ „์ฒ˜๋ฆฌ


ย ์ด๋ฒˆ์— ์ €๋Š” ๋ฐ˜์ง€์˜ ์ œ์™• ๋ชจ๋ธ์ฒ˜๋Ÿผ LoveCraft ์Šคํƒ€์ผ์˜ ๊ธ€์„ ๋งŒ๋“ค์–ด๋‚ด๋Š” ๋ชจ๋ธ์„ ๋งŒ๋“ค๊ณ  ์‹ถ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๊ธฐ ์œ„ํ•ด์„  ์ž‘์—…ํ•ด์•ผ ํ•  ๊ฒƒ์ด ์ข€ ๋งŽ์Šต๋‹ˆ๋‹ค. ํŒŒ์ผ๋ณ„๋กœ ์ œ๋ชฉ๊ณผ ์ฑ•ํ„ฐํ‘œ๊ธฐ๊ฐ€ ์žˆ๊ณ , ์ผ๋ถ€ ๊ณต๋ฐฑ์ด NBSP๋กœ ๋˜์–ด์žˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, ๊ธ€์„ ๋ณด๊ธฐ ์ข‹๊ฒŒํ•˜๊ธฐ ์œ„ํ•ด ๊ฐœํ–‰๋“ค์ด ๋นˆ๋ฒˆํ•˜๊ฒŒ ์‚ฝ์ž…๋˜์–ด์žˆ์Šต๋‹ˆ๋‹ค. concat.txt ๋ฅผ ์‚ฌ์šฉํ•  ์ˆœ ์žˆ์ง€๋งŒ, concat.txt๋Š” ์ฑ•ํ„ฐ ํ‘œ๊ธฐ๋‚˜ ๋จธ๋ฆฌ๋ง, ์ธ์šฉ๊ตฌ ๊ฐ™์€ ์ œ๊ฐ€ ํ•„์š”ํ•˜์ง€ ์•Š์€ ๊ฒƒ๋“ค์ด ์žˆ์–ด ์‚ฌ์šฉํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค.
ย ๋จผ์ €, ๋ฐ์ดํ„ฐ๋ฅผ ํ•ฉ์น˜๊ธฐ ์ „์— chapter ํ‘œ๊ธฐ๋ฅผ ์†์ˆ˜ ์ง€์›Œ์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค. ์ •๊ทœ์‹์œผ๋กœ ์ œ๊ฑฐํ•˜๋Š” ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•  ์ˆ˜ ์žˆ์ง€๋งŒ, ํŒŒ์ผ๋งˆ๋‹ค ์ฑ•ํ„ฐ ํ‘œ๊ธฐ ๋ฐฉ์‹์ด ๋‹ค์†Œ ์ฐจ์ด๊ฐ€ ์žˆ์–ด๋ณด์—ฌ ์ง์ ‘ ์ง€์› ์Šต๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ, ์ง€์šฐ๋ฉด์„œ ๋ณด๋‹ˆ ์ผ์ •ํ•œ ํฌ๋งท์ด ์žˆ๋Š” ๊ฒƒ ๊ฐ™์•„ ์—ฌ๋Ÿฌ๋ถ„์ด ํ•˜์‹ค ๋•Œ๋Š” ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•˜๋Š” ๊ฒƒ์ด ํŽธํ• ๊ฒ๋‹ˆ๋‹ค.
with open(result_dir + '/' + result_name + '.txt', 'w', encoding='utf-8-sig') as r:

    for file in file_list:
        name = file.split('.')

        if 'txt' not in name:
            continue

        with open(source_dir + '/' + file, 'r', encoding='utf-8') as f:
            lines = f.readlines()
						
	    # ์ œ๋ชฉ ํ–‰์„ ์นด์šดํŠธํ•˜๊ธฐ ์œ„ํ•œ ๋ณ€์ˆ˜์ž…๋‹ˆ๋‹ค
            count = 0

            for line in lines:
		# 0๋ฒˆ ์งธ ์ค„์€ ์ œ๋ชฉ์ž…๋‹ˆ๋‹ค.
		# ๋”ฐ๋ผ์„œ, 0๋ฒˆ ์ค„์€ ์ ์ง€์•Š๊ณ  ๊ฑด๋„ˆ ๋œ๋‹ˆ๋‹ค.
                if count == 0:
                    count = 1
                    continue

                r.write(line + " ")

ย ํŒŒ์ผ์„ ํ•ฉ์น˜๋Š” ๊ฐ„๋‹จํ•œ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค. ๋ช…๋ น์–ด(ex. ๋ฆฌ๋ˆ…์Šค์˜ cat *txt > text.txt)๋ฅผ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ์ง€๋งŒ, ์ œ๋ชฉ๊นŒ์ง€ ์‚ญ์ œํ•˜๊ธฐ ์œ„ํ•ด ์ฝ”๋“œ๋กœ ์ž‘์„ฑํ–ˆ์Šต๋‹ˆ๋‹ค.
๋ฐ์ดํ„ฐ๋ฅผ ํ•˜๋‚˜๋กœ ๋งŒ๋“ค์—ˆ์œผ๋‹ˆ ์ด์ œ ์ „์ฒด์ ์œผ๋กœ ์†์„ ๋ด์ฃผ๋ฉด GPT-2๋ฅผ ํ•™์Šต์‹œํ‚ค๊ธฐ ์œ„ํ•œ ๋ฐ์ดํ„ฐ๊ฐ€ ์™„์„ฑ๋  ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค!

with open(source_dir + '/' + text_file, 'r', encoding='utf-8') as f:

    name = text_file.split('.')[0]
    # ์ด ๋‹จ์–ด๋“ค์€ ๋ฌธ์žฅ์ด ๋์ด ์•„๋‹Œ ๊ฒƒ๋“ค ์ž…๋‹ˆ๋‹ค.
    whos = ['Dr\\.$', 'Mr\\.$', 'Ms\\.$', 'Mrs\\.$', '\\.\\.+$']

    with open(result_dir + '/' + name + '.txt', 'w', encoding='utf-8') as r:
        lines = f.readlines()

        for line in lines:
	    # ์•„๋ฌด ๊ธ€์ด ์—†๋Š” ๋ผ์ธ์€ ๊ฑด๋„ˆ๋œ๋‹ˆ๋‹ค.
            if line in ["\\n", "", " ", " \\n"] or not line:
                continue

	    # nbsp๋กœ ๋˜์–ด์žˆ๋Š” ๊ณต๋ฐฑ์„ ์ผ๋ฐ˜์ ์ธ ๊ณต๋ฐฑ์œผ๋กœ ๋ฐ”๊ฟ‰๋‹ˆ๋‹ค
            line = unidecode.unidecode(line)
	    # ๊ณต๋ฐฑ๋“ค์„ ์ •๊ทœํ™”ํ•ด์ค๋‹ˆ๋‹ค.
            text = " ".join(line.split())
	    # ... ๋“ค์ด . . . ์œผ๋กœ ๋˜์–ด์žˆ์–ด ...์œผ๋กœ ๋ฐ”๊ฟ”์ฃผ๋Š” ๋ผ์ธ์ž…๋‹ˆ๋‹ค.
            text = re.sub("\\s\\.", ".", text)

	    # ์•„๋ฌด ๊ธ€์ด ์—†๋Š” ํ…์ŠคํŠธ
            if len(text) == 0:
                continue
	    # ๋ผ์ธ์ด .์œผ๋กœ ๋๋‚ฌ๋‹ค๋ฉด, ๋ฌธ์žฅ์ด ๋๋‚œ ๊ฒƒ์œผ๋กœ ๊ฐ„์ฃผํ•ฉ๋‹ˆ๋‹ค.
            elif text[-1] in ["."]:
	    # .์œผ๋กœ ๋๋‚ฌ์–ด๋„ whos์— ํฌํ•จ๋œ ๊ฒƒ์€ ๋ฌธ์žฅ์˜ ๋์ด ์•„๋‹™๋‹ˆ๋‹ค.
                for who in whos:
                    if re.search(who, text):
		        # ๋ฌธ์žฅ์˜ ๋์ด ์•„๋‹ˆ๊ธฐ ๋•Œ๋ฌธ์— ๊ณต๋ฐฑ์„ ๋„ฃ์–ด์ค๋‹ˆ๋‹ค.
                        text += " "
                        break
		# for-else๋ฌธ
                else:
                    text += "\\n"
	    # ๋ฌธ์žฅ์˜ ๋์ด ์•„๋‹ˆ๊ธฐ ๋•Œ๋ฌธ์— ๊ณต๋ฐฑ์„ ๋„ฃ์–ด์ค๋‹ˆ๋‹ค.
            else:
                text = " " + text

            r.write(text)

ย ํ•ฉ์ณ์ง„ ๋ฐ์ดํ„ฐ๋ฅผ ์ „์ฒ˜๋ฆฌํ•˜๋Š” ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค. ๋ถˆํ•„์š”ํ•œ ๊ฐœํ–‰์„ ์ค„์ด๊ณ  ๋ฌธ๋‹จ๋ณ„๋กœ ๋ฌถ์–ด์ฃผ๊ธฐ ์ฃผ๋Š” ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ๋งค๋ฒˆ ํ•˜๋Š” ์ž‘์—…์ด์ง€๋งŒ, ๊ผญ ํ•„์š”ํ•œ ์ƒํ™ฉ(ex. ์‹œ, ๋…ธ๋ž˜๋ฅผ ๋งŒ๋“œ๋Š” ๋ชจ๋ธ)์ด ์•„๋‹ˆ๋ผ๋ฉด ์ œ๊ฑฐํ•˜๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.

 "Heave to, there's something floating to the leeward" the speaker was a short stockily built man whose name was William Jones. he was the captain of a small cat boat in whichhe & a party of men were sailing at the time the story opens.
 "Aye aye sir" answered John Towers & the boat was brought to a stand still Captain Jones reached out his hand for the object which he now discerned to be a glass bottle "Nothing but a rum flask that the men on a passing boat threw over" he said but from an impulse of curiosity he reached out for it. it was a rum flask & he was about to throw it away when he noticed a piece of paper in it. He pulled it out & on it read the following Jan 1 1864 I am John Jones who writes this letter my ship is fast sinking with a treasure on board I am where it is marked * on the enclosed chart Captain Jones turned the sheet over & the other side was a chart on the edge were written these words dotted lines represent course we took "Towers" Said Capt. Jones exitedly "read this" Towers did as he was directed "I think it would pay to go" said Capt. Jones "do you"? "Just as you say" replied Towers.
...

ย 6292 line์˜ 3.9mb ๋ฐ์ดํ„ฐ๊ฐ€ ์™„์„ฑ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ์ด์ œ ์ด๊ฑธ๋กœ ๊ณตํฌ์Šค๋Ÿฌ์šด LoveCraft ๋ชจ๋ธ์„ ๋งŒ๋“ค์–ด ๋ด…์‹œ๋‹ค!


3. ํ•™์Šต

๏ปฟ
๏ปฟ ์ด๋ฒˆ์—๋Š” ModelType์„ small๋กœ ํ•˜๊ณ  epoch๋Š” 3์œผ๋กœ ํ•˜์˜€์Šต๋‹ˆ๋‹ค. ์ด์ „๊นŒ์ง„ epoch๋ฅผ ๊ณ„์† 5๋ฅผ ์‚ฌ์šฉํ–ˆ๋Š”๋ฐ, ๋ฐ˜๋“œ์‹œ 5๊ฐ€ ์•„๋‹ˆ๋ผ๋„ ์ข‹์€ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ค๋Š”์ง€ ์•Œ๊ณ ์‹ถ์–ด 3์„ ์‚ฌ์šฉํ•ด ๋ณด์•˜์Šต๋‹ˆ๋‹ค.

๏ปฟ
๏ปฟ
์ด๋ฒˆ์—๋Š” ModelType์„ small๋กœ ํ•˜๊ณ  epoch๋Š” 3์œผ๋กœ ํ•˜์˜€์Šต๋‹ˆ๋‹ค. ์ด์ „๊นŒ์ง„ epoch๋ฅผ ๊ณ„์† 5๋ฅผ ์‚ฌ์šฉํ–ˆ๋Š”๋ฐ, ๋ฐ˜๋“œ์‹œ 5๊ฐ€ ์•„๋‹ˆ๋ผ๋„ ์ข‹์€ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ค๋Š”์ง€ ์•Œ๊ณ ์‹ถ์–ด 3์„ ์‚ฌ์šฉํ•ด ๋ณด์•˜์Šต๋‹ˆ๋‹ค.


4. ์ƒ์„ฑ

๏ปฟ
๏ปฟํ•™์Šต์ด ๋๋‚ฌ์Šต๋‹ˆ๋‹ค! View API๋ฅผ ํ†ตํ•ด Ainize์— ๋นŒ๋“œ๋œ ์ œ ๋ชจ๋ธ์„ ๋ณผ ์ˆ˜ ์žˆ๊ณ , Test your model์„ ํ†ตํ•ด Tabtab ์—์„œ ์ œ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•ด ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

 **From the dark, I saw** the thin lines of its carvings... those damnable circles, those perfect circles...." My thirst had nearly caked me, yet I kept not running. The idea of beholding a vast subterrene world surging through the air of subterrene loathsomeness horrified and unnerved me, but my curiosity about the whole business was magnified a thousandfold. Surely, the Old Ones must be close to the visible universe, yet I could not vouch for their absolute perfection. For in addition to their uncanny light and shadow, there was something else vastly more--something infinitely more--unrelatively similar in mentality and construction.
 It was I who thought of the blasphemous and unholy carvings on those frightful carvings; the monstrously resemblances which the carversation of those carvings seemed to imply. That their presence was preternatural, I could from a logical and scientific point of view admit; but it was more absolutely distinct from the vague pseudo-memories of other beings--others than those which Carter had dreamt about. The vague pseudo-memories were, I felt, all connected with the one thing I knew--the door of that room with the carven carven carven face and carven face. I recalled once talking with a human being of that race--being guessed from olden letters--who lived in that room, and who said they had talked with strange and significant beings of other universes. And of that being I could speak with a written tongue, though it was alien to me.

Tabtab์—์„œ ๋ชจ๋ธ์„ ํ…Œ์ŠคํŠธํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค. "From the dark, I saw" ๋ฅผ ์ž…๋ ฅ์œผ๋กœ ์ฃผ๊ณ  ์ƒ์„ฑ๋œ ๊ฒฐ๊ณผ์ธ๋ฐ, ๊ธฐ๋Œ€ํ•œ ๊ฒƒ ์ฒ˜๋Ÿผ ์ƒ๋‹นํžˆ ์œผ์Šค์Šคํ•œ ๊ธ€์ด ๋‚˜์™”์Šต๋‹ˆ๋‹ค. epoch๋ฅผ 3๋งŒ ์ฃผ์—ˆ๋Š”๋ฐ๋„ ์ƒ๋‹นํžˆ ์ข‹์€ ๊ฒฐ๊ณผ์ž…๋‹ˆ๋‹ค. ๋นจ๋ฆฌ ๊ฒฐ๊ณผ๋ฅผ ๋ณด๊ณ  ์‹ถ์œผ์‹  ๋ถ„์€ epoch๋ฅผ ์ ๊ฒŒ ์ฃผ์–ด๋„ ๊ดœ์ฐฎ์„ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค!
์ €๋Š” small๊ณผ 3 epoch๋ฅผ ์‚ฌ์šฉํ•ด๋ณด์•˜์ง€๋งŒ, ์—ฌ๋Ÿฌ๋ถ„์€ ๋‹ค์–‘ํ•œ ์กฐํ•ฉ์œผ๋กœ ๋ชจ๋ธ์„ ๋งŒ๋“ค์–ด ๋ณด์„ธ์š”. ๋‹ค์–‘ํ•˜๊ณ  ์žฌ๋ฐŒ๋Š” ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ฌ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค. Teachable NLP๋ฅผ ๋‹ค์–‘ํ•˜๊ฒŒ ์ฆ๊ฒจ๋ณด์„ธ์š”!
Like Comment

Teachable NLP๋กœ ๋™ํ™” ์ž‘์„ฑ NLP ๋ชจ๋ธ ๋งŒ๋“ค๊ธฐ

Teachable NLP: link
Tabtab: link

์ด์ „์—๋Š” ๋ฐ˜์ง€์˜ ์ œ์™• ํ…์ŠคํŠธ ๋ฐ์ดํ„ฐ๋ฅผ ์ด์šฉํ•ด Teachable NLP๋กœ GPT-2 ๋ชจ๋ธ์„ ํ•™์Šต์„ ์‹œ์ผฐ์—ˆ์Šต๋‹ˆ๋‹ค. > link
ย Kaggle์—์„œ ํฅ๋ฏธ๋กœ์šด ๋ฐ์ดํ„ฐ๋ฅผ ์ฐพ์•˜์Šต๋‹ˆ๋‹ค. ๋ฐ”๋กœ ์—ฌ๋Ÿฌ ๋™ํ™”๋“ค์ด ๋ชจ์•„์ ธ์žˆ๋Š” ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ์ด ๋ฐ์ดํ„ฐ๋Š” CC0: Public Domain๋กœ ๋ˆ„๊ตฌ๋‚˜ ์ €์ž‘๊ถŒ ์ œํ•œ์ด ์—†์–ด ์ž์œ ๋กญ๊ฒŒ ์‚ฌ์šฉํ•  ์ˆ˜ ์žˆ๋Š” ๋ผ์ด์„ผ์Šค์ธ๋ฐ, ์ž์„ธํ•œ ๊ฑด ๋งํฌ๋ฅผ ํ†ตํ•ด ํ™•์ธํ•ด ๋ณด์‹œ๋Š” ๊ฒƒ์„ ์ถ”์ฒœ๋“œ๋ฆฝ๋‹ˆ๋‹ค. ์ด๋ฒˆ์—” ์ด๊ฑธ๋กœ GPT-2 ๋ชจ๋ธ์„ ํ•™์Šต์‹œ์ผœ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

1. ๋ฐ์ดํ„ฐ


Lovely IlonkaThere was once a kingโ€™s son who told his father that he wished to marry.โ€˜No, no!โ€™ said the king; โ€˜you must not be in such a hurry. Wait till you
have done some great deed. My father did not let me marry till I had won
the golden sword you see me wear.โ€™The prince was much disappointed, but he never dreamed of disobeying his
father, and he began to think with all his might what he could do. It
was no use staying at home, so one day he wandered out into the world to
try his luck, and as he walked along he came to a little hut in which he
found an old woman crouching over the fire.โ€˜Good evening, mother. I see you have lived long in this world; do you
know anything about the three bulrushes?โ€™โ€˜Yes, indeed, Iโ€™ve lived long and been much about in the world, but I
have never seen or heard anything of what you ask. Still, if you will
wait till to-morrow I may be able to tell you something.โ€™
โ€ฆ

ย ์ด๊ฒƒ์ด ์ด๋ฒˆ์— ์‚ฌ์šฉํ•  ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค. ๋™ํ™”์˜ ์ œ๋ชฉ๊ณผ ๊ณต๋ฐฑ์„ ํฌํ•จํ•ด์„œ 246991 ๋ผ์ธ์˜ 12.5mb์˜ ์—„์ฒญ๋‚œ ํ…์ŠคํŠธ ํŒŒ์ผ์ž…๋‹ˆ๋‹ค!! ์ข‹์€ ๋ฐ์ดํ„ฐ์ด์ง€๋งŒ, ์ผ๋ถ€๋ถ„์€ ์ „์ฒ˜๋ฆฌ๋ฅผ ํ•ด์•ผ ํ•  ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.
ย ๊ณต๋ฐฑ์€ ํ•™์Šต์— ๋„์›€์ด ๋˜์ง€ ์•Š๊ณ  ํ•„์š” ์—†๋Š” ๊ฐœํ–‰๋„ ๋นˆ๋ฒˆํ•ฉ๋‹ˆ๋‹ค. ์ด ๋‘˜์€ ๋ชจ๋ธ์ด ์ƒ์„ฑํ•ด๋‚ด๋Š” ๊ธ€์„ ์–ด์ƒ‰ํ•˜๊ฒŒ ๋งŒ๋“ค ์ˆ˜ ์žˆ์œผ๋‹ˆ ์ œ๊ฑฐํ•˜์˜€์Šต๋‹ˆ๋‹ค.
๋ฐ์ดํ„ฐ์— ๋™ํ™”์˜ ์ œ๋ชฉ์ด ํฌํ•จ๋˜์–ด ์žˆ์ง€๋งŒ, ์ด๋ฒˆ์—๋Š” ์ œ๊ฑฐํ•˜์ง€ ์•Š๊ณ  ํŠน์ˆ˜ํ•˜๊ฒŒ ์‚ฌ์šฉํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. ์ œ๋ชฉ์— ํŠน์ˆ˜ํ•œ ์ฒ˜๋ฆฌ๋ฅผ ํ•ด์„œ ์ œ๋ชฉ์„ ์ž…๋ ฅํ•˜๋ฉด, ๋ชจ๋ธ์ด ๊ทธ ์ œ๋ชฉ์„ ๋ณด๊ณ  ๋™ํ™”๋ฅผ ์ƒ์„ฑํ•ด๋‚ด๊ฒŒ ๋งŒ๋“ค๊ฒŒ ํ•ด๋ณผ ์ƒ๊ฐ์ž…๋‹ˆ๋‹ค.
ย ๊ทธ๋Ÿผ ๊ทธ ํŠน์ˆ˜ํ•œ ์ฒ˜๋ฆฌ๋Š” ๋ฌด์—‡์ด๋ƒ! ๋ฐ”๋กœ ๋ชจ๋ธ์ด ์ œ๋ชฉ์„ ๊ตฌ๋ณ„ํ•  ์ˆ˜ ์žˆ๊ฒŒ ํ† ํฐ์„ ์ฃผ์–ด๋ณผ ๊ฒƒ์ž…๋‹ˆ๋‹ค. ํ˜„์žฌ Teachable NLP๋Š” ์ปค์Šคํ…€ ํ† ํฐ์„ ์ถ”๊ฐ€ํ•˜๋Š” ๊ธฐ๋Šฅ์€ ์—†์ง€๋งŒ, ๊ฐ„๋‹จํ•œ ๋ฐฉ๋ฒ•์œผ๋กœ ๋น„์Šทํ•œ ๊ธฐ๋Šฅ์ด ๋˜๊ฒŒ ๋งŒ๋“ค ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

with open(source_dir + '/' + text_file, 'r', encoding='utf-8') as f:

    name = text_file.split('.')[0]

    with open(result_dir + '/' + name + '.txt', 'w', encoding='utf-8') as r:
        lines = f.readlines()

	# ์•„๋ฌด ๊ธ€์ด ์—†๋Š” ๋ผ์ธ์„ ์นด์šดํŠธํ•˜๋Š” ๋ณ€์ˆ˜์ž…๋‹ˆ๋‹ค
	# ๋ฐ์ดํ„ฐ์˜ ์ฒซ ์ค„๋„ ์ œ๋ชฉ์ด๋ฏ€๋กœ ์กฐ๊ฑด๋ฌธ์— ๊ฑธ๋ฆด ์ˆ˜ ์žˆ๊ฒŒ 5๋กœ ์„ค์ •ํ–ˆ์Šต๋‹ˆ๋‹ค.
        count = 5

        for line in lines:
	    # ์•„๋ฌด ๊ธ€์ด ์—†๋Š” ๋ผ์ธ์„ ์ œ๊ฑฐํ•˜๋Š” ๋ถ€๋ถ„์ž…๋‹ˆ๋‹ค. 
            if line == "\n" or line == "" or line == " ":
                count += 1
                continue

	    # ๋ฐ์ดํ„ฐ๋ฅผ ๋ณด๊ณ  ํŒ๋‹จํ•œ ๊ฒฐ๊ณผ๋กœ,
	    # ์นด์šดํŠธ๊ฐ€ 3๊ฐœ ์ด์ƒ ์Œ“์ด๋ฉด ์ƒˆ๋กœ์šด ๋™ํ™”๊ฐ€ ์‹œ์ž‘๋˜๋Š” ๊ฒƒ์œผ๋กœ ๊ฐ„์ฃผํ•ฉ๋‹ˆ๋‹ค.
		# ๋™ํ™”์˜ ์ฒซ ์ค„์€ ๋ฐ˜๋“œ์‹œ ์ œ๋ชฉ๋ผ์ธ ์ž…๋‹ˆ๋‹ค.
            elif count > 3:
                count = 0
		# ๋ผ์ธ์˜ ๊ณต๋ฐฑ์„ ์ •๊ทœํ™” ํ•˜๋Š” ๋ถ€๋ถ„์ž…๋‹ˆ๋‹ค.
		# ์ด ๋ฐฉ๋ฒ•์„ ์‚ฌ์šฉํ•˜๋ฉด ์—ฐ์†๋œ ๊ณต๋ฐฑ์€ ๋ฌผ๋ก , ๊ฐœํ–‰ ๊นŒ์ง€ ์ œ๊ฑฐ๋ฉ๋‹ˆ๋‹ค.
                text = " ".join(line.split())

		# ์ œ๋ชฉ ์•ž๋’ค๋กœ <title>, </title>์„ ๋„ฃ์–ด์ค๋‹ˆ๋‹ค.
		# ์ด๊ฒƒ์ด ์ œ๋ชฉ์„ ๊ตฌ๋ถ„ํ•˜๋Š” ํ† ํฐ ์—ญํ• ์„ ํ•ด์ค„ ๊ฒƒ ์ž…๋‹ˆ๋‹ค.
                r.write("\n<title>" + text + "</title>")
                continue

	    # ์นด์šดํŠธ๊ฐ€ 3์ด ๋˜์ง€ ์•Š์•˜์ง€๋งŒ, 1 ์ด์ƒ์ด๋ผ๋ฉด
	    # ํ•œ ๋ฌธ๋‹จ์ด ๋๋‚œ ๊ฒƒ์ด๋ผ ๊ฐ„์ฃผํ•˜๊ณ  ๊ฐœํ–‰์„ ํ•ด์ค๋‹ˆ๋‹ค.
            if count > 0:
                r.write("\n ")
            count = 0

	    # ๋ผ์ธ์˜ ๊ณต๋ฐฑ์„ ์ •๊ทœํ™” ํ•˜๋Š” ๋ถ€๋ถ„์ž…๋‹ˆ๋‹ค.
            text = " ".join(line.split())

	    # ๋งˆ์ง€๋ง‰์œผ๋กœ, ์™„์„ฑ๋œ ๋ผ์ธ์„ ์ ์–ด์ค๋‹ˆ๋‹ค.
	    # ๊ฐœํ–‰ ๋Œ€์‹  ๊ณต๋ฐฑ์„ ๋„ฃ์–ด ์ž์—ฐ์Šค๋Ÿฌ์šด ๋ฌธ์žฅ์ด ๋˜๊ฒŒ ํ•ด์ค๋‹ˆ๋‹ค.
            r.write(text + " ")    


ย ๋™ํ™” ๋ฐ์ดํ„ฐ๋ฅผ ์ „์ฒ˜๋ฆฌํ•˜๊ธฐ ์œ„ํ•œ ๊ฐ„๋‹จํ•œ ํŒŒ์ด์ฌ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค. ์กฐ๊ฑด๋ฌธ์œผ๋กœ ๋ถˆํ•„์š”ํ•œ ๋ผ์ธ์„ ์ œ๊ฑฐํ•˜๊ณ  ์ œ๋ชฉ ๋ผ์ธ์— <title></title>์ด๋ผ๋Š” ๊ตฌ๋ถ„ ํ† ํฐ์„ ์ถ”๊ฐ€ํ–ˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, ๋ถˆํ•„์š”ํ•œ ๊ฐœํ–‰๋“ค์„ ์ค„์—ฌ ๋ฌธ๋‹จ๋ณ„๋กœ ํ•˜๋‚˜์˜ ๋ผ์ธ์œผ๋กœ ๋งŒ๋“ค์–ด ์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค.

<title>Lovely Ilonka</title>
There was once a kingโ€™s son who told his father that he wished to marry.
โ€˜No, no!โ€™ said the king; โ€˜you must not be in such a hurry. Wait till you have done some great deed. My father did not let me marry till I had won the golden sword you see me wear.โ€™
The prince was much disappointed, but he never dreamed of disobeying his father, and he began to think with all his might what he could do. It was no use staying at home, so one day he wandered out into the world to try his luck, and as he walked along he came to a little hut in which he found an old woman crouching over the fire.
โ€˜Good evening, mother. I see you have lived long in this world; do you know anything about the three bulrushes?โ€™
โ€˜Yes, indeed, Iโ€™ve lived long and been much about in the world, but I have never seen or heard anything of what you ask. Still, if you will wait till to-morrow I may be able to tell you something.โ€™
Well, he waited till the morning, and quite early the old woman appeared and took out a little pipe and blew in it, and in a moment all the crows in the world were flying about her. Not one was missing. Then she asked if they knew anything about the three bulrushes, but not one of them did.
The prince went on his way, and a little further on he found another hut in which lived an old man. On being questioned the old man said he knew nothing, but begged the prince to stay overnight, and the next morning the old man called all the ravens together, but they too had nothing to tell.
The prince bade him farewell and set out. He wandered so far that he crossed seven kingdoms, and at last, one evening, he came to a little house in which was an old woman.
โ€ฆ

ย ์ „์ฒ˜๋ฆฌ๊ฐ€ ์™„๋ฃŒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค! 41882 ๋ผ์ธ์˜ 12.5mb์˜ ๋ฉ‹์ง„ ๋ฐ์ดํ„ฐ๊ฐ€ ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ์ด๊ฑธ๋กœ GPT-2 ๋ชจ๋ธ์„ fine-tuning ํ•˜๋ฉด ๋ฉ‹์ง„ ๋™ํ™” ์ƒ์„ฑ๊ธฐ๊ฐ€ ๋  ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค!

2. Training

ย 
ย ์ด๋ฒˆ์—” ModelType์€ small(GPT-2 small)์„ ์‚ฌ์šฉํ•˜๊ณ  epoch๋Š” 5๋ฅผ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค. small ๋ชจ๋ธ์€ ์ ์€ ๋ฐ์ดํ„ฐ๋กœ ํŠœ๋‹ํ•˜๋”๋ผ๋„ ๋‹ค์–‘ํ•œ ๊ฒฐ๊ณผ๋ฅผ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋Š” ํ›Œ๋ฅญํ•œ ๋ชจ๋ธ์ด๊ธฐ ๋•Œ๋ฌธ์— ์‚ฌ์šฉํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค.
ย Train Model ๋ฒ„ํŠผ์„ ๋ˆ„๋ฅด๋‹ˆ 1์‹œ๊ฐ„ ์ •๋„ ์†Œ์š”๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ๋‹คํ–‰ํžˆ Teachable NLP๋Š” ์ฐฝ์„ ๊บผ๋„ ๊ณ„์† ์ž‘์—…์ด ์ง„ํ–‰๋˜๊ธฐ ๋•Œ๋ฌธ์— ๊ทธ๋™์•ˆ AI Network Forum์—์„œ ๋‹ค๋ฅธ ๋ถ„๋“ค์ด ์˜ฌ๋ ค๋†“์€ Use cases๋ฅผ ๋‘˜๋Ÿฌ๋ณด๊ฑฐ๋‚˜ Ainize์˜ ํ”„๋กœ์ ํŠธ๋“ค์„ ์‚ฌ์šฉํ•ด๋ณด๋Š” ๋“ฑ, ๋‹ค์–‘ํ•œ ์ผ์„ ํ•  ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.

3. ์ƒ์„ฑ


<title>In my dream</title>
From the Danish.
There was once a man who had three sons, and they were all married, and very proud.
The youngest of the three princes was named Martin, for he was clever and agreeable, and besides, he was also a courageous youth, and knew how to make a voyage. So he said, โ€˜I am going to take a voyage to some great country, and you shall go with me.โ€™
โ€˜Do not trouble yourself about that,โ€™ said the elder; โ€˜the journey is worth doing, and it will bring you luck.โ€™
So he walked on with his brothers across the sea to the town where their father had lived before they were married. Then they entered a beautiful house, and the young man said, โ€˜Now listen; you went out into the world, and what do you find in it?โ€™
โ€˜Alas!โ€™ said he, โ€˜I left home to seek my fortune, and now I have found it! Here I must stay till two beautiful days, and then the three days are just as full of joy as ever.โ€™
โ€˜But what happens after that?โ€™ asked the younger brother, wondering at his brotherโ€™s strange history. โ€˜Oh, well! I am going to set out on the journey, and I will go on with you.โ€™
And so he walked on with his brothers across the sea to the land of the little town where they had lived before they were married. The elder brother thought that he would like to stay with him, for he could accompany him on the journey, but the younger one wished to go on with him, for nothing.
When they reached home and saw his sister, who was much farther off than the other two days, and she was much more beautiful, and also younger, he said to himself, 'Why, but he would like to travel too. So he said to travel too, and he said, but he did not like it. So he travelled and he stayed at once.

ย Tabtab์—์„œ "<title>In my dream</title>"์„ ์ž…๋ ฅ์œผ๋กœ ์ฃผ๊ณ  GPT-2 ๋ชจ๋ธ์ด ์ƒ์„ฑํ•ด๋‚ธ ๊ฒฐ๊ณผ์ž…๋‹ˆ๋‹ค. ์ œ๋ชฉ๋งŒ ์ž…๋ ฅํ–ˆ๋Š”๋ฐ๋„ ์„ธ ํ˜•์ œ๋“ค์— ๊ด€ํ•œ ๊ทธ๋Ÿด์‹ธํ•œ ์ด์•ผ๊ธฐ๊ฐ€ ๋งŒ๋“ค์–ด์กŒ์Šต๋‹ˆ๋‹ค. ์—ฌ๋Ÿฌ๋ถ„๋„ ์ œ๊ฐ€ ํ•™์Šต์‹œํ‚จ ๋ชจ๋ธ์„ ํ•œ๋ฒˆ ์‚ฌ์šฉํ•ด๋ณด์„ธ์š” ๐Ÿ˜€
Like Comment

Teachable NLP์™€ ํ•จ๊ป˜ ๋ฐ˜์ง€์˜ ์ œ์™• ๊ฐ๋ณธ์„ ์จ๋ณด์ž! ๐Ÿ’

Teachable NLP: link
Tabtab: link

ย Teachable-nlp๋Š” GPT-2์„ ๋ณต์žกํ•œ ์ฝ”๋“œ ์—†์ด! GPU(๊ทธ๋ž˜ํ”ฝ์นด๋“œ) ํ•„์š” ์—†์ด! ํŠœ๋‹์„ ํ•  ์ˆ˜ ์žˆ๊ฒŒ ๋„์™€์ฃผ๋Š” no-code ํ”„๋กœ๊ทธ๋žจ์ž…๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ GPT-2 ๋ชจ๋ธ์€ ๋‹ค์–‘ํ•œ ๊ธ€์„ ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ๋Š” transformers ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค. ์ด ๋ชจ๋ธ์€ ์–ด๋–ค ๋ฐ์ดํ„ฐ๋กœ ํ•™์Šต์‹œํ‚ค๋Š๋ƒ์— ๋”ฐ๋ผ ์†Œ์„ค, ์‹œ, ๋…ธ๋ž˜ ๊ฐ™์€ ๊ฑธ ์ž‘์„ฑํ•  ์ˆ˜ ์žˆ๋Š” ๋งค๋ ฅ์ ์ธ ๋ชจ๋ธ๋กœ ๋ณ€์‹ ํ•ฉ๋‹ˆ๋‹ค.
ย ๊ทธ๋Ÿผ teachable-nlp์— ๋ฌด์—‡์„ training ์‹œ์ผœ๋ณผ๊นŒ ๊ณ ๋ฏผํ•˜๋˜ ๋„์ค‘, ์„ธ๊ณ„์ ์œผ๋กœ ์œ ๋ช…ํ•œ ํ†จํ‚จ์˜ ์†Œ์„ค ๋ฐ˜์ง€์˜ ์ œ์™•์ด ๋– ์˜ฌ๋ž๊ณ , ๋ฐ˜์ง€์˜ ์ œ์™• ํŠธ๋ฆด๋กค๋กœ์ง€์— ๋Œ€ํ•œ ๋ฐ์ดํ„ฐ๋ฅผ Kaggle์—์„œ ์ฐพ์„ ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.


1. ๋ฐ์ดํ„ฐ!


THE RETURN OF THE KING
       _being the third part of
       The Lord of the Rings_

                      _Chapter 1_
       Minas Tirith

Pippin looked out from the shelter of Gandalf's cloak. He wondered if he was awake or still sleeping, still in the swift-moving dream in which he had been wrapped so long since the great ride began. The dark world was rushing by and the wind sang loudly in his ears. He could see nothing but the wheeling stars, and away to his right vast shadows against the sky where the mountains of the South marched past. Sleepily he tried to reckon the times and stages of their journey, but his memory was drowsy and uncertain.
There had been the first ride at terrible speed without a halt, and then in the dawn he had seen a pale gleam of gold, and they had come to the silent town and the great empty house on the hill. And hardly had they reached its shelter when the winged shadow had passed over once again, and men wilted with fear. But Gandalf had spoken soft words to him, and he had slept in a corner, tired but uneasy, dimly aware of comings and goings and of men talking and Gandalf giving orders. And then again riding, riding in the night. This was the second, no, the third night since he had looked in the Stone. And with that hideous memory he woke fully, and shivered, and the noise of the wind became filled with menacing voices.
A light kindled in the sky, a blaze of yellow fire behind dark barriers Pippin cowered back, afraid for a moment, wondering into what dreadful country Gandalf was bearing him. He rubbed his eyes, and then he saw that it was the moon rising above the eastern shadows, now almost at the full. So the night was not yet old and for hours the dark journey would go on. He stirred and spoke.
'Where are we, Gandalf?' he asked.
'In the realm of Gondor,' the wizard answered. 'The land of An๏ฟฝrien is still passing by.'

โ€ฆ

ย ๋ฐ˜์ง€์˜ ์ œ์™• ๋ฐ์ดํ„ฐ๋Š” 3๊ฐœ๋กœ ๋‚˜๋ˆ ์ ธ ์žˆ์—ˆ๊ณ , ๋งŽ์€ ์•…์„ผํŠธ ๋ฌธ์ž๋“ค์ด ๏ฟฝ๋กœ ๊นจ์ ธ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.
์ œ๊ฐ€ ์›ํ•˜๋Š” ๊ฒฐ๊ณผ๋Š” ๋ฐ˜์ง€์˜ ์ œ์™• ์†Œ์„ค ๋Š๋‚Œ์˜ ๊ธ€์„ ์ƒ์„ฑํ•˜๋Š” ๋ชจ๋ธ์„ ๋งŒ๋“ค๊ณ  ์‹ถ์€๋ฐ, ๋ฐ์ดํ„ฐ๋ฅผ ๊ทธ๋Œ€๋กœ ๋„ฃ์œผ๋ฉด ๋ถˆํ•„์š”ํ•œ ๊ณต๋ฐฑ๊ณผ ๋‹จ์–ด๊ฐ€ ํ•™์Šต ๋  ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ๋ถˆํ•„์š”ํ•œ ๊ฒƒ๋“ค์„ ์ •๋ฆฌํ•˜๊ธฐ๋กœ ํ–ˆ์Šต๋‹ˆ๋‹ค.


2. ์ „์ฒ˜๋ฆฌ!

๏ปฟ
๏ปฟ ์ €๋Š” ์ „์ฒ˜๋ฆฌ๋ฅผ ์œ„ํ•ด Notepad++๋ฅผ ์‚ฌ์šฉํ•˜์˜€์Šต๋‹ˆ๋‹ค. Notepad++๋Š” ํ…์ŠคํŠธ ํŒŒ์ผ์„ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ๋Š” ๋‹ค์–‘ํ•œ ๊ธฐ๋Šฅ๋“ค์„ ์ œ๊ณตํ•˜๋Š” ํ”„๋ฆฌ์›จ์–ด์ž…๋‹ˆ๋‹ค. ์ด ํ”„๋กœ๊ทธ๋žจ์ด ์ œ๊ณตํ•˜๋Š” ๊ธฐ๋Šฅ๋“ค์„ ์ด์šฉํ•ด ํ…์ŠคํŠธ ํŒŒ์ผ์„ ์ œ๊ฐ€ ์›ํ•˜๋Š” ๋ชจ์Šต์œผ๋กœ ๋ฐ”๊ฟ”๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ๋จผ์ €, ์ด ํŒŒ์ผ์€ ๊ธ€์„ ๋ฌธ๋‹จ๋ณ„๋กœ ์ •๋ฆฌํ•˜๊ธฐ ์œ„ํ•ด ๊ณต๋ฐฑ๋“ค์ด ๋งŽ์ด ์กด์žฌํ•ฉ๋‹ˆ๋‹ค. ์ด๋Ÿฐ ์ž˜๋ชป๋œ ๊ณต๋ฐฑ์„ ํ•™์Šตํ•˜๋ฉด ์ƒ์„ฑ ๋œ ๊ฒฐ๊ณผ์— ์–ด์ƒ‰ํ•œ ๊ณต๋ฐฑ์ด ๋‚˜ํƒ€๋‚  ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์Šต๋‹ˆ๋‹ค. ํ•„์š”์—†๋Š” ๊ณต๋ฐฑ๋“ค์„ ์ œ๊ฑฐํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค. ํŽธ์ง‘ > ๊ณต๋ฐฑ ๊ธฐ๋Šฅ > ์ค„ ์‹œ์ž‘/๋ ๊ณต๋ฐฑ ์ œ๊ฑฐ ๋ฅผ ์„ ํƒํ•˜์—ฌ ์ค„ ์•ž, ๋’ค์— ์žˆ๋Š” ๊ณต๋ฐฑ๋“ค์„ ์ œ๊ฑฐํ•ด ์ค๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ๊ณต๋ฐฑ์ด ์ค„์–ด๋“  ๋•๋ถ„์— ํ•œ๊ฒฐ ๋ณด๊ธฐ ์ข‹์•„์ง€๊ณ  ๋‹ค๋ฃจ๊ธฐ ํŽธํ•ด์กŒ์Šต๋‹ˆ๋‹ค. ์ด์ œ ๋‹ค์Œ ์Šคํ…์œผ๋กœ ๋„˜์–ด๊ฐ€ ๋ด…์‹œ๋‹ค!ย 

Three Rings for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.

FOREWORD

...

PROLOGUE

...

----------------------------------------------------------

THE FELLOWSHIP OF THE RING

_being the first part of
The Lord of the Rings_

BOOK I

...

========================================================

ย ๋‹ค์Œ์œผ๋กœ ๊ฐ€์žฅ ๋ˆˆ์— ๋„๋Š” ๊ฒƒ์€ ํ•„์š” ์—†์„ ๊ฒƒ ๊ฐ™์€ ๋จธ๋ฆฌ๋ง๊ณผ ์ฑ…์†Œ๊ฐœ ๊ทธ๋ฆฌ๊ณ  ์ฑ…์˜ ๋งˆ์ง€๋ง‰ ๋ถ€๋ถ„ ํ‘œ์‹œ ์ž…๋‹ˆ๋‹ค. ์ด ๋ถ€๋ถ„์„ ํ•™์Šต์— ์‚ฌ์šฉํ•˜๋ฉด ์ƒ์„ฑ๋œ ์†Œ์„ค์˜ ํ๋ฆ„์ด ๋Š์–ด ์ง€๊ฑฐ๋‚˜ ์–ด์ƒ‰ํ•œ ๋‹จ์–ด๊ฐ€ ๋‚˜์˜ฌ ๊ฒƒ ๊ฐ™์•˜์Šต๋‹ˆ๋‹ค. ํ•˜์ง€๋งŒ, ์ด๊ฒƒ๋“ค์€ ์ž๋™์œผ๋กœ ์ œ๊ฑฐํ•˜๋Š” ๋ฐฉ๋ฒ•์€ ์—†์œผ๋ฏ€๋กœ ์†์ˆ˜ ์ง€์›Œ์ฃผ์—ˆ์Šต๋‹ˆ๋‹ค. ๋‹จ, PROLOGUE์˜ ๋‚ด์šฉ์€ ๋ฐ˜์ง€์˜ ์ œ์™•์˜ ๋ฐฉ๋Œ€ํ•œ ๋‚ด์šฉ ์ค‘ ์ผ๋ถ€์ด๋ฏ€๋กœ ์ง€์šฐ๋ฉด ์•ˆ ๋ฉ๋‹ˆ๋‹ค!ย 

_Chapter 1_
A Long-expected Party
ย ์–ด๋Š์ •๋„ ์ฒ˜๋ฆฌ๋ฅผ ํ–ˆ์ง€๋งŒ, ์•„์ง ๋ถ€์กฑํ•ฉ๋‹ˆ๋‹ค. Chapter ํ‘œ๊ธฐ๋“ค์ด ๋‚จ์•„์žˆ์Šต๋‹ˆ๋‹ค. ์ด ํ‘œ๊ธฐ๋“ค์€ ์ฑ…์„ ์ฝ์„ ๋•Œ, ๋…์ž๋“ค์—๊ฒŒ ์ด์•ผ๊ธฐ์˜ ํ๋ฆ„์„ ์•Œ๋ ค์ฃผ๋Š” ๊ธธ์žก์ด์ง€๋งŒ, ํŠน๋ณ„ํ•˜๊ฒŒ ์‚ฌ์šฉํ•˜์ง€ ์•Š๋Š”๋‹ค๋ฉด ๋ชจ๋ธ Training์—๋Š” ๋ฐฉํ•ด๊ฐ€ ๋  ๋ฟ์ž…๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ์ฑ•ํ„ฐ ํ‘œ๊ธฐ๋“ค์„ ์ง€์šฐ๊ธฐ ์œ„ํ•ด ์ƒ๋‹จ๋ฐ”์˜ ์ฐพ๊ธฐ > ์ฑ…๊ฐˆํ”ผ๋ฅผ ์„ ํƒํ•ฉ๋‹ˆ๋‹ค. ๋‹จ์ถ•ํ‚ค๋Š” Ctrl + M ์ž…๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ๊ทธ๋Ÿผ ์ด๋Ÿฐ ์ฐฝ์ด ๋‚˜ํƒ€๋‚˜๊ฒŒ ๋ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ ์ค„์„ ์ฑ…๊ฐˆํ”ผ๋ฅผ ์ฒดํฌํ•œ ํ›„, ์ฐพ์„ ๋‚ด์šฉ์— _Chapter๋ฅผ ์ž…๋ ฅํ•ด ์ค๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋ฉด, _Chapter์ด ์žˆ๋Š” ์ค„ ์•ž์— ์ฑ…๊ฐˆํ”ผ ํ‘œ์‹œ์ธ โ—์ด ํ‘œ๊ธฐ๋ฉ๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟChapter ๋ผ์ธ์€ ์ฑ…๊ฐˆํ”ผ ํ‘œ๊ธฐ๊ฐ€ ๋˜์—ˆ์ง€๋งŒ, ์ œ๋ชฉ์€ ํ‘œ๊ธฐ ๋˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค. ์ฐพ๊ธฐ๋กœ _Chapter๋ฅผ ๊ฒ€์ƒ‰ํ•ด Chapter์™€ ์ œ๋ชฉ๋“ค์„ ์ฐพ์€ ๋’ค, ์ œ๋ชฉ๋ผ์ธ์˜ ์•ž์„ ํด๋ฆญํ•ด ์ „๋ถ€ ์ฑ…๊ฐˆํ”ผ ํ‘œ๊ธฐ๋ฅผ ํ•ด์ค๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ๊ทธ๋ฆฌ๊ณ  ์ƒ๋‹จ๋ฐ”์˜ ์ฐพ๊ธฐ > ์ฑ…๊ฐˆํ”ผ > ์ฑ…๊ฐˆํ”ผ ๋œ ์ค„ ์‚ญ์ œํ•˜๊ธฐ ๋ฅผ ์‚ฌ์šฉํ•˜๋ฉด... ์งœ์ž”! ์ฑ•ํ„ฐ์™€ ์ œ๋ชฉ๋“ค์ด ๋ชจ๋‘ ์‚ญ์ œ๋˜์—ˆ์Šต๋‹ˆ๋‹ค! ๋ฐ”๋กœ ๊ฒ€์ƒ‰ํ•ด์„œ ์ง€์šฐ๋Š” ๊ฒƒ์ด ๋” ํŽธํ•  ์ˆ˜๋Š” ์žˆ์ง€๋งŒ, ์œ ์šฉํ•œ ๊ธฐ๋Šฅ์ด๋ผ ์†Œ๊ฐœํ•ด๋“œ๋ ธ์Šต๋‹ˆ๋‹ค.
๏ปฟ
๏ปฟ ๋งˆ์ง€๋ง‰์œผ๋กœ ์•„๋ฌด๊ฒƒ๋„ ์—†๋Š” ๋ผ์ธ๋“ค๋งŒ ์ œ๊ฑฐํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค! ๋นˆ ๋ผ์ธ๋„ ์ž˜๋ชป๋œ ๊ณต๋ฐฑ๋“ค ์ฒ˜๋Ÿผ ๋ชจ๋ธ ํ•™์Šต์„ ๋ฐฉํ•ดํ•˜์—ฌ ์–ด์ƒ‰ํ•œ ๊ฒฐ๊ณผ๋ฅผ ๋งŒ๋“ค์–ด๋‚ผ ๊ฐ€๋Šฅ์„ฑ์ด ์žˆ์Šต๋‹ˆ๋‹ค. ์ƒ๋‹จ๋ฐ”์˜ ํŽธ์ง‘ > ์ค„๋‹จ์œ„ ํŽธ์ง‘ > ๋นˆ ์ค„ ์ œ๊ฑฐ (๊ณต๋ฐฑ ๋ฌธ์ž ํฌํ•จ) ์„ ๋ˆ„๋ฅด๋ฉด ์†์‰ฝ๊ฒŒ ๋นˆ ๋ผ์ธ๋“ค์„ ์ œ๊ฑฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!
์œ„ ๊ณผ์ •๋“ค์„ ๋‹ค๋ฅธ ๋‘ ํŒŒ์ผ์—๋„ ํ•ด์ค€ ๋’ค, ํŒŒ์ผ์„ ํ•ฉ์ณ์ฃผ๊ธฐ๋งŒ ํ•˜๋ฉด ๋์ž…๋‹ˆ๋‹ค. Ctrl + A ๋ฅผ ๋ˆŒ๋Ÿฌ ๋ชจ๋“  ํ…์ŠคํŠธ๋ฅผ ์„ ํƒํ•œ ๋’ค ๋ณต์‚ฌํ•˜์—ฌ ๋ถ™ํ˜€๋„ฃ๊ธฐ๋งŒ ํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค!

Hobbits are an unobtrusive but very ancient people, more numerous formerly than they are today; for they love peace and quiet and good tilled earth: a well-ordered and well-farmed countryside was their favourite haunt. They do not and did not understand or like machines more complicated than a forge-bellows, a water-mill, or a hand-loom, though they were skilful with tools. Even in ancient days they were, as a rule, shy of โ€˜the Big Folkโ€™, as they call us, and now they avoid us with dismay and are becoming hard to find. They are quick of hearing and sharp-eyed, and though they are inclined to be fat and do not hurry unnecessarily, they are nonetheless nimble and deft in their movements. They possessed from the first the art of disappearing swiftly and silently, when large folk whom they do not wish to meet come blundering by; and this an they have developed until to Men it may seem magical.ย 
โ€ฆ

ย 754 line์„ ๊ฐ€์ง„ 2.5mb์˜ text ๋ฐ์ดํ„ฐ๋ฅผ ๋งŒ๋“ค์–ด๋ƒˆ๊ณ , ํŠœ๋‹์— ์‚ฌ์šฉํ•  ์ค€๋น„๋œ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค! 1 mb ์ •๋„์˜ ๋ฐ์ดํ„ฐ๋งŒ ์žˆ์–ด๋„ GPT-2๋ฅผ fine-tuning ํ•˜๊ธฐ์—” ์ถฉ๋ถ„ํ•ฉ๋‹ˆ๋‹ค.


2. Training!


ย Teachable-nlp์— ๋ฐ์ดํ„ฐ๋ฅผ ์—…๋กœ๋“œํ•˜๊ณ  ํŠœ๋‹์„ ํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค. ์ฒ˜์Œ์—” large(GPT-2 large ๋ชจ๋ธ)์„ ์‚ฌ์šฉํ•ด๋ณด๊ณ  ์‹ถ์—ˆ์ง€๋งŒ, ์‹œ๊ฐ„์ด ์˜ค๋ž˜ ๊ฑธ๋ฆด ๊ฒƒ ๊ฐ™์•„ ModelType์€ medium (GPT-2 medium ๋ชจ๋ธ)์„ ์‚ฌ์šฉํ•˜๊ณ  epoch๋Š” ์ข‹์€ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ค๊ธธ ํฌ๋งํ•˜๋ฉฐ 5๋ฅผ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค. medium๋„ ์‹œ๊ฐ„์ด ๋งŽ์ด ๊ฑธ๋ฆด ๊ฒƒ ๊ฐ™์•˜์ง€๋งŒ, ํŠœ๋‹์‹œ๊ฐ„์€ 30๋ถ„ ์ •๋„๋กœ ์งง๊ฒŒ ์†Œ์š”๋์Šต๋‹ˆ๋‹ค.

3. ์ƒ์„ฑ!


The hobbits started heading towards the tower of ๏ฟฝ๏ฟฝowyn, as soon as the gates were shut again.
They found her still clinging to the lex in her hurry. โ€˜I am not a Black Rider, ๏ฟฝomer,โ€™ she said.

ย Training์ด ์™„๋ฃŒ๋œ ํ›„, Test your model์„ ๋ˆ„๋ฅด๊ณ  Tabtab ์—์„œ ์™„์„ฑ๋œ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•ด ๋ณด์•˜์Šต๋‹ˆ๋‹ค.
ย ํฅ๋ฏธ๋กœ์›Œ ๋ณด์ด๋Š” ๋ฐ˜์ง€์˜ ์ œ์™• ์Šคํƒ€์ผ์˜ ๊ธ€์ด ์ƒ์„ฑ๋˜์—ˆ์ง€๋งŒ, ์ผ๋ถ€ ๊ธ€์ž๋“ค์ด ๏ฟฝ๋กœ ๋˜์–ด์žˆ์–ด ๋ณด๊ธฐ๊ฐ€ ์•ˆ ์ข‹์•„๋ณด์ž…๋‹ˆ๋‹ค.

4. ๋‹ค์‹œ Training!


ย ๏ฟฝ๋“ค์„ ์›๋ž˜ ๊ธ€์ž๋กœ ๋ณต๊ตฌ์‹œํ‚ค๋Š” ๋ฐฉ๋ฒ•์„ ๋ฌผ์ƒ‰๋ณด์•˜์ง€๋งŒ, ๊ทธ๋Ÿฐ ๋ชจ๋“ˆ์ด๋‚˜ ๋ฐฉ๋ฒ•์€ ์—†์–ด ๋ณด์˜€๊ณ  ๊ทธ๋ ‡๋‹ค๊ณ  ์ง€์›Œ๋ฒ„๋ฆฌ๋ฉด ์ œ๋Œ€๋กœ๋œ ๋ฐ˜์ง€์˜ ์ œ์™•๊ธ€์ด ๋˜์ง€ ์•Š์„ ๊ฒƒ ๊ฐ™์•„ ๋‹จ์–ด๋“ค์˜ ์›๋ž˜ ๊ธ€์ž๊ฐ€ ๋ญ”์ง€ ๊นจ์ง„ ๋‹จ์–ด๋“ค์„ ๊ตฌ๊ธ€๋งํ•˜์—ฌ ์œ ์ถ”ํ•˜์—ฌ ์ฐพ์•„๋ณด๊ธฐ๋กœ ํ•˜์˜€์Šต๋‹ˆ๋‹ค.
๏ปฟ
Fix_list.txt
4.27 KB
๏ปฟ
์ธํ„ฐ๋„ท์—์„œ ์ฐพ์•„๋‚ธ ์›๋ž˜ ๊ธ€์ž๋“ค ๋ชจ์€ ํŒŒ์ผ์ž…๋‹ˆ๋‹ค. ๊นจ์ง„ ๋ฌธ์ž๋“ค์ด ์ƒ๋‹นํžˆ ๋งŽ์Šต๋‹ˆ๋‹ค. ์ด๊ฒƒ๋“ค๋„ Notepad++์˜ ๊ธ€์ž ๋ฐ”๊พธ๊ธฐ ๊ธฐ๋Šฅ์œผ๋กœ ์›๋ž˜ ๊ต์ •ํ•ด์ค๋‹ˆ๋‹ค. ์ด ๋ถ€๋ถ„์€ ๋ฉ”๋ชจ์žฅ, ๊ตฌ๊ธ€ docx, vi ํŽธ์ง‘๊ธฐ ๋“ฑ ๊ฐ์ข… ํŽธ์ง‘๊ธฐ ํ”„๋กœ๊ทธ๋žจ๋“ค์ด ์ œ๊ณตํ•˜๋Š” ๊ธฐ๋Šฅ์„ ์‚ฌ์šฉํ•˜์—ฌ ๋ฌธ์ž๋“ค์„ ๊ต์ •ํ•˜๊ณ  ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ๊ณ , ํŒŒ์ด์ฌ์˜ replace๊ฐ™์€ ์ฝ”๋“œ๋ฅผ ์ด์šฉํ•ด ๋ฐ”๊ฟ€ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.ย 

ย ๋‹ค์‹œ ํ•œ ๋ฒˆ Teachable-nlp์— ๋ฐ์ดํ„ฐ๋ฅผ ์—…๋กœ๋“œํ•˜๊ณ  ํŠœ๋‹์„ ํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค. ์ด๋ฒˆ์—๋„ ModelType์€ medium์„ ์‚ฌ์šฉํ•˜๊ณ  epoch๋Š” 5๋ฅผ ์‚ฌ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค.


5. ๋‹ค์‹œ ์ƒ์„ฑ!


The hobbits started heading towards the tower of The eastern window of the inn was open, and a crowd was there waiting for them inside.
โ€˜Be careful of your words, Sam!โ€™ said Strider in a low voice. 'I have not seen the Lady of the Galadhrim in seven days, nor spoken with the Lady again.
Some have accused me of sorcery, but I have not studied it.

ย ๋‹ค์‹œ ํ•œ ๋ฒˆ Tabtab ์—์„œ ์™„์„ฑ๋œ ๋ชจ๋ธ์„ ์‚ฌ์šฉํ•ด ๋ณด์•˜์Šต๋‹ˆ๋‹ค. ๊นจ์ง„ ๊ธ€์ž๋“ค์ด ๊ต์ •๋˜์–ด ์ข€ ๋” ๊ทธ๋Ÿด๋“ฏํ•œ ๊ฒฐ๊ณผ๊ฐ€ ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
ย ๊ฒฐ๊ณผ๊ฐ€ ์–ด๋–ค๊ฐ€์š”? ๊ทธ๋Ÿด๋“ฏํ•ด ๋ณด์ด์ง€ ์•Š๋‚˜์š”? ์—ฌ๋Ÿฌ๋ถ„๋„ ์ œ๊ฐ€ ๋งŒ๋“  ๋ชจ๋ธ์„ ํ•œ ๋ฒˆ ์‚ฌ์šฉํ•ด ๋ณด์…”์„œ ์—ฌ๋Ÿฌ๋ถ„ ๋งŒ์˜ ๋ฐ˜์ง€์˜ ์ œ์™•์„ ๋งŒ๋“ค์–ด ๋ณด์„ธ์š”.
๊ทธ๋ฆฌ๊ณ  ์—ฌ๋Ÿฌ๋ถ„๋“ค๋„ ์—ฌ๋Ÿฌ๋ถ„์ด ์ข‹์•„ํ•˜๋Š” ๊ธ€์„ ์ด์šฉํ•ด์„œ ๋ชจ๋ธ์„ ๋งŒ๋“ค์–ด๋ณด์„ธ์š”. ์žฌ๋ฏธ๋‚œ ํŒฌํ”ฝ์„ ๋งŒ๋“ค ์ˆ˜ ์žˆ๋Š” ๋ชจ๋ธ์„ ๋งŒ๋“ค ์ˆ˜ ์žˆ์„ ๊ฒ๋‹ˆ๋‹ค!
Like Comment